{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 第八章 注意力机制\n",
    "\n",
    "\n",
    "注意力机制（Attention Mechanism）是目前在深度学习中使用非常多的信息选择机制。注意力机制可以作为一种资源分配方案，将从大量的候选信息中选择和任务更相关或更重要的信息，是解决信息超载问题的有效手段。注意力机制可以单独使用，但是更多地用作神经网络中的一个组件。\n",
    "\n",
    "本章内容基于《神经网络与深度学习》第8章：注意力机制相关内容进行设计。在阅读本章之前，建议先了解如图8.1所示的关键知识点，以便更好的理解和掌握相应的理论和实践知识。\n",
    "\n",
    "本章内容主要包含两部分：\n",
    "\n",
    "* **模型解读**：实现注意力机制的基本模式和其变体，并设计了一个文本分类的实验，验证通过注意力机制和LSTM模型的组合应用，验证对模型处理信息能力的提升效果；并进一步实现多头自注意力模型来实现文本分类。\n",
    "\n",
    "* **案例与实践**：实现基于Transformer模型的文本语义匹配任务。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/843904864e314fc099c2eb38ff26447f6197aab319ca4c82a6de2d4cae2279a5\" width=\"600px\"></center>\n",
    "<br><center>图8.1 注意力机制关键知识点回顾</center></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 8.1. 基于双向LSTM和注意力机制的文本分类\n",
    "\n",
    "\n",
    "注意力机制的计算可以分为两步：一是在所有序列元素上计算注意力分布，二是根据注意力分布来计算序列中所有元素表示的加权平均得到的聚合表示。\n",
    "\n",
    "为了从$N$个**输入向量**$[x_{1};...;x_{N}]$中选择出和某个特定任务相关的信息，需要引入一个和任务相关的表示，称为**查询向量**{Query Vector}，并通过一个打分函数来计算每个输入向量和查询向量之间的相关性。\n",
    "\n",
    "给定一个和任务相关的查询向量$q$，首先计算**注意力分布**{Attention Distribution}，即选择第$n$个输入向量的概率$\\alpha_{n}$：\n",
    "\n",
    "$$\n",
    "\\alpha_{n}=softmax(s(\\mathbf x_{n},\\mathbf q))\n",
    "$$\n",
    "\n",
    "其中$s(x,q)$为**注意力打分函数**，可以使用**加性模型**、**点积模型**、**缩放点积模型**和**双线性模型**的方式来计算。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "得到注意力分布之后，可以对输入向量进行加权平均，得到整个序列的最终表示。\n",
    "$$\n",
    "\\mathbf z = \\sum_{n=1}^{N}\\alpha_{n} \\mathbf x_n.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "在第6.4节中的文本分类任务中，经过双向LSTM层之后，直接将序列中所有元素的表示进行平均，得到的平均表示作为整个输入序列的聚合表示。这种平均的聚合方式虽然达到不错的效果，但是还不够精确。\n",
    "在本节中，我们首先在第6.4节中实现的基于LSTM网络进行文本分类任务的基础上，通过在LSTM层上再叠加一层的注意力机制来从LSTM的隐状态中的自动选择有用的信息。\n",
    "\n",
    "\n",
    "如图8.2所示，输入一个文本序列The movie is nice进行情感分析，直观上单词nice应该比其它词更重要。这时可以利用注意力机制来挑选对任务更相关的信息。假设给定一个和任务相关的查询向量为sentiment，我们先用查询向量sentiment和文本序列中所有词计算注意力分布，并根据注意力分布对所有词进行加权平均，得到整个序列的聚合表示。\n",
    "\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/eaf583596e934cc8a266a42b998356453d7241c05a774c7f89e828d23ca31093\" width=\"800px\"></center>\n",
    "<br><center>图8.2 使用注意力机制来计算序列的聚合表示示例</center></br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.1.1 数据\n",
    "\n",
    "\n",
    "\n",
    "本实验使用和第6.4.1节相同的数据集：IMDB电影评论数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data:\n",
      "Text: it does seem like this film is polarizing us. you either love it or hate it. i loved it.<br /><br />i agree with the comment(s) that said, you just gotta \"feel\" this one.<br /><br />also, early in the film, tom cruise shows his girlfriend a painting done by monet--an impressionist painter. monet's style is to paint in little dabs so up close the painting looks like a mess, but from a distance, you can tell what the subject is. cruise mentions that the painting has a \"vanilla sky\". i believe this is a hint to the moviegoer. this movie is like that impressionist painting. it's impressionist filmmaking! and it's no coincidence that the title of the movie refers to that painting.<br /><br />this is not your typical linear plot. it requires more thought. there is symbolism and there are scenes that jump around and no, you're not always going to be sure what's going on. but at the end, all is explained.<br /><br />you will need to concentrate on this movie but i think people are making the mistake of concentrating way too hard on it. after it ends is when you should think about it. if you try to figure it out as it's unfolding, you will overwhelm yourself. just let it happen...\"go\" with it...keep an open mind. remember what you see and save the analysis for later.<br /><br />i found all the performances top notch and thought it to be tremendously unique, wildly creative, and spellbinding.<br /><br />but i will not critize the intelligence of those of you who didn't enjoy it. it appeals to a certain taste. if you like existential, psychedelic, philosophical, thought-provoking, challenging, spiritual movies, then see it. if you prefer something a little lighter, then skip it.<br /><br />but if you do like what i described, then you will surely enjoy it.; Label 1\n"
     ]
    }
   ],
   "source": [
    "from data import load_vocab,load_imdb_data\n",
    "train_data, dev_data, test_data = load_imdb_data(\"./dataset\") # 加载IMDB数据集和word2id词典\n",
    "word2id_dict= load_vocab(\"dataset/vocab.txt\") # 加载词典\n",
    "# 显示一条训练样本\n",
    "print(\"Train data:\")\n",
    "text,label=train_data[0]\n",
    "print(f\"Text: {text}; Label {label}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集样本数： 25000\n",
      "处理后的第一条样本： ([12, 114, 268, 37, 10, 24, 7, 84022, 2440, 25, 452, 120, 12, 40, 779, 112, 9, 403, 619, 13, 165, 1041, 17, 2, 146466, 11, 1100, 25, 39, 3633, 28663, 10, 2506, 13, 4158, 358, 8, 2, 151, 877, 4597, 287, 21, 1384, 3, 4644, 247, 31, 146467, 25606, 26222, 146468, 507, 7, 6, 2858, 8, 105, 146469, 38, 65, 547, 2, 4644, 251, 37, 3, 5877, 18, 34, 3, 19243, 25, 64, 337, 48, 2, 961, 701, 4597, 5165, 11, 2, 4644, 41, 3, 63486, 60235, 9, 250, 10, 7, 3, 3275, 6, 2, 63493, 10, 20, 7, 37, 11, 25606, 15503, 44, 25606, 146470, 4, 44, 60, 9045, 11, 2, 490, 5, 2, 20, 6415, 6, 11, 46562, 13, 255, 7, 23, 111, 716, 9899, 1076, 12, 3616, 51, 5492, 52, 7, 5302, 4, 52, 22, 159, 11, 2051, 197, 4, 1375, 298, 23, 206, 162, 6, 28, 274, 777, 162, 774, 18, 29, 2, 656, 35, 7, 26138, 13, 2230, 74, 311, 6, 6864, 19, 10, 20, 18, 9, 98, 89, 22, 242, 2, 1887, 5, 14977, 108, 104, 261, 19, 112, 94, 12, 654, 7, 50, 25, 128, 98, 43, 112, 46, 25, 325, 6, 848, 12, 47, 15, 44, 47240, 25, 74, 14975, 3782, 39, 347, 12, 146471, 17, 146472, 32, 990, 1954, 366, 48, 25, 67, 4, 544, 2, 7625, 16, 9074, 13, 165, 236, 35, 2, 423, 426, 5339, 4, 199, 12, 6, 28, 8730, 7704, 5944, 14683, 4, 146473, 13, 688, 9, 74, 23, 146474, 2, 2199], 1)\n"
     ]
    }
   ],
   "source": [
    "from nndl import IMDBDataset\n",
    "\n",
    "batch_size = 128\n",
    "max_seq_len = 256\n",
    "train_set = IMDBDataset(train_data, word2id_dict, max_seq_len)\n",
    "dev_set = IMDBDataset(dev_data, word2id_dict, max_seq_len)\n",
    "test_set = IMDBDataset(test_data, word2id_dict, max_seq_len)\n",
    "\n",
    "print('训练集样本数：', len(train_set))\n",
    "print('处理后的第一条样本：', train_set[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[Tensor(shape=[128, 256], dtype=int64, place=Place(gpu:0), stop_gradient=True,\n",
      "       [[12    , 114   , 268   , ..., 146474, 2     , 2199  ],\n",
      "        [10    , 20    , 7     , ..., 1     , 1     , 1     ],\n",
      "        [9     , 98    , 11    , ..., 31    , 2798  , 3398  ],\n",
      "        ...,\n",
      "        [9     , 309   , 6     , ..., 13    , 4328  , 99    ],\n",
      "        [10506 , 11342 , 41    , ..., 10    , 160   , 77    ],\n",
      "        [10    , 14    , 30    , ..., 1     , 1     , 1     ]]), Tensor(shape=[128], dtype=int64, place=Place(gpu:0), stop_gradient=True,\n",
      "       [256, 198, 256, 256, 174, 174, 256, 150, 132, 256, 128, 102, 123, 79 ,\n",
      "        171, 115, 254, 122, 149, 117, 130, 256, 116, 212, 89 , 177, 194, 113,\n",
      "        123, 179, 135, 49 , 194, 256, 125, 91 , 256, 256, 92 , 128, 218, 137,\n",
      "        256, 256, 255, 173, 145, 116, 45 , 167, 256, 154, 129, 76 , 131, 157,\n",
      "        193, 134, 226, 256, 256, 122, 256, 151, 256, 156, 192, 197, 256, 173,\n",
      "        71 , 256, 105, 55 , 141, 163, 256, 256, 142, 256, 41 , 93 , 110, 78 ,\n",
      "        60 , 256, 133, 256, 256, 256, 256, 28 , 256, 190, 256, 256, 115, 145,\n",
      "        256, 61 , 134, 151, 123, 134, 135, 149, 256, 256, 256, 256, 204, 47 ,\n",
      "        124, 88 , 176, 83 , 256, 129, 187, 73 , 80 , 93 , 256, 256, 208, 256,\n",
      "        256, 45 ])], Tensor(shape=[128, 1], dtype=int64, place=Place(gpu:0), stop_gradient=True,\n",
      "       [[1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [0],\n",
      "        [1]])]\n"
     ]
    }
   ],
   "source": [
    "from functools import partial\n",
    "import paddle\n",
    "from paddle.io import DataLoader\n",
    "\n",
    "def collate_fn(batch_data, pad_val=1):\n",
    "    seqs, labels, lens = [], [], []\n",
    "    for seq, label in batch_data:\n",
    "        seqs.append(seq)\n",
    "        labels.append([label])\n",
    "        lens.append(len(seq))\n",
    "    \n",
    "    max_len = max(lens)\n",
    "    for i in range(len(seqs)):\n",
    "        seqs[i] = seqs[i] + [pad_val] * (max_len - len(seqs[i]))\n",
    "    \n",
    "    return (paddle.to_tensor(seqs), paddle.to_tensor(lens)),paddle.to_tensor(labels)\n",
    "\n",
    "\n",
    "train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=False, drop_last=False, collate_fn=collate_fn)\n",
    "dev_loader = DataLoader(dev_set, batch_size=batch_size, shuffle=False, drop_last=False, collate_fn=collate_fn)\n",
    "test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False, drop_last=False, collate_fn=collate_fn)\n",
    "\n",
    "for ix,  batch in enumerate(train_loader):\n",
    "    if ix == 0:\n",
    "        print(batch)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.1.2 模型构建\n",
    "\n",
    "本实验的模型结构如图所示。整个模型由以下几个部分组成：\n",
    "\n",
    "1） 嵌入层：将输入句子中的词语转换为向量表示；\n",
    "\n",
    "2） LSTM层：基于双向LSTM网络来建模句子中词语的上下文表示；\n",
    "\n",
    "3） 注意力层：使用注意力机制来从LSTM层的输出中筛选和聚合有效的特征；\n",
    "\n",
    "4） 线性层：输出层，预测对应的类别得分。\n",
    "\n",
    "我们直接使用第6.4节中实现的嵌入层和双向LSTM层，这里主要介绍注意力层的实现。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/1849e001101c4b8695889a1eb0c6f82368d467c0a2434780aa1f2314cb1c525f\" width=\"800px\"></center>\n",
    "<br><center>图8.3 基于双向LSTM和注意力机制的文本分类模型</center></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "\n",
    "假设在给定一个长度为$L$的序列，在经过嵌入层和双向LSTM层之后，我们得到序列的表示矩阵$\\mathbf X\\in \\mathbb{R}^{L \\times D}$，其中$D$为序列中每个元素的特征表示维度。\n",
    "在本节中，我们利用注意力机制来进行更好的信息聚合，只需要从$\\mathbf X$中选择一些和任务相关的信息作为序列的聚合表示。\n",
    "\n",
    "下面我们分别实现在注意力计算中的注意力打分函数、注意力分布计算和加权平均三个模块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.2.1 注意力打分函数\n",
    "\n",
    "首先我们实现注意力分布计算公式8.1节中的注意力打分函数$s(\\mathbf x,\\mathbf q)$。\n",
    "这里，我们分别实现**加性模型**和**点积模型**两种。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**加性模型**\n",
    "\n",
    "\n",
    "假设输入序列为$\\mathbb X\\in \\mathbb{R}^{B\\times L\\times D}$，其中$B$为批量大小，$L$为序列长度，$D$为特征维度，我们引入一个任务相关的查询向量$\\mathbb q\\in \\mathbb{R}^{D}$，这里查询向量$\\mathbb q$作为可学习的参数。\n",
    "\n",
    "加性模型的公式为\n",
    "$$\n",
    "s(\\mathbf X,\\mathbf q)=\\mathbf v^T \\tanh(\\mathbf X\\mathbf W+\\mathbf{q}^T \\mathbf U),\n",
    "$$\n",
    "其中$\\mathbf W\\in \\mathbb{R}^{D\\times D}$,$\\mathbf U\\in \\mathbb{R}^{D\\times D}$和$\\mathbf v\\in \\mathbb{R}^{D}$都是可学习的参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "********\n",
    " 在本章中，我们实现形如$\\mathbf X\\mathbf W$这种特征矩阵和参数矩阵的乘积时，可以直接利用`paddle.nn.Linear()`算子来实现，这样可以使得实现代码更简洁。\n",
    "  $\\mathbf Y=\\mathbf X\\mathbf W$可以实现为：\n",
    "\n",
    "    W = paddle.nn.Linear(D, D, bias_attr=False)\n",
    "    Y = W(X)\n",
    "********"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0714 14:14:08.415232   103 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 8.0, Driver API Version: 11.2, Runtime API Version: 11.2\n",
      "W0714 14:14:08.419399   103 gpu_context.cc:306] device: 0, cuDNN Version: 8.2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(shape=[1, 3], dtype=float32, place=Place(gpu:0), stop_gradient=False,\n",
      "       [[-0.52282858,  0.84343088,  0.59816718]])\n"
     ]
    }
   ],
   "source": [
    "import paddle.nn as nn\n",
    "import paddle\n",
    "\n",
    "class AdditiveScore(nn.Layer):\n",
    "    def __init__(self, hidden_size):\n",
    "        super(AdditiveScore, self).__init__()\n",
    "        self.fc_W = nn.Linear(hidden_size, hidden_size, bias_attr=False)\n",
    "        self.fc_U = nn.Linear(hidden_size, hidden_size, bias_attr=False)\n",
    "        self.fc_v = nn.Linear(hidden_size, 1, bias_attr=False)\n",
    "        # 查询向量使用均匀分布随机初始化\n",
    "        self.q = paddle.create_parameter(\n",
    "            shape=[1, hidden_size],\n",
    "            dtype=\"float32\",\n",
    "            default_initializer=nn.initializer.Uniform(low=-0.5, high=0.5),\n",
    "        )\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - inputs：输入矩阵，shape=[batch_size, seq_len, hidden_size]\n",
    "        输出：\n",
    "            - scores：输出矩阵，shape=[batch_size, seq_len]\n",
    "        \"\"\"\n",
    "        # inputs:  [batch_size, seq_len, hidden_size]\n",
    "        batch_size, seq_len, hidden_size = inputs.shape\n",
    "        # scores: [batch_size, seq_len, hidden_size]\n",
    "        scores = paddle.tanh(self.fc_W(inputs)+self.fc_U(self.q))\n",
    "        # scores: [batch_size, seq_len]\n",
    "        scores = self.fc_v(scores).squeeze(-1)\n",
    "        return scores\n",
    "\n",
    "paddle.seed(2021)\n",
    "inputs = paddle.rand(shape=[1, 3, 3])\n",
    "additiveScore = AdditiveScore(hidden_size=3)\n",
    "scores = additiveScore(inputs)\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上面显示的是加性注意力打分算子的打分输出，加性注意力打分算子的输入是随机初始化的张量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    " ********\n",
    " 这里加性模型是按照《神经网络与深度学习》的公式(8.2)来实现。由于在本任务中，$\\mathbf{q}$也作为可学习的参数，因此$\\mathbf{q}^T \\mathbf{U}$也可以简化为一组参数$\\mathbf{q}$。请思考两种实现方式的区别？\n",
    " ********"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**点积模型**\n",
    "\n",
    "下面我们再来实现点积的注意力模型。\n",
    "对于输入序列为$\\mathbf X\\in \\mathbb{R}^{B\\times L\\times D}$，其中$B$为批量大小，$L$为序列长度，$D$为特征维度，以及可学习的任务相关的查询向量$\\mathbf q\\in \\mathbb{R}^{D}$，点积模型的公式为\n",
    "$$\n",
    "s(\\mathbf X,\\mathbf q)=\\mathbf X\\mathbf q.\n",
    "$$\n",
    "其中$\\mathbf q$是一个和任务相关的可学习的查询向量。理论上，加性模型和点积模型的复杂度差不多，但是点积模型在实现上可以更好地利用矩阵乘积，从而计算效率更高。\n",
    "\n",
    "将点积模型实现为**点积注意力打分算子**，代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(shape=[1, 3], dtype=float32, place=Place(gpu:0), stop_gradient=False,\n",
      "       [[-0.24780354,  0.31022751,  0.19821185]])\n"
     ]
    }
   ],
   "source": [
    "class DotProductScore(nn.Layer):\n",
    "    def __init__(self, hidden_size):\n",
    "        super(DotProductScore, self).__init__()\n",
    "        # 使用均匀分布随机初始化一个查询向量\n",
    "        self.q = paddle.create_parameter(\n",
    "            shape=[hidden_size, 1],\n",
    "            dtype=\"float32\",\n",
    "            default_initializer=nn.initializer.Uniform(low=-0.5, high=0.5),\n",
    "        )\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - X：输入矩阵，shape=[batch_size,seq_len,hidden_size]\n",
    "        输出：\n",
    "            - scores：输出矩阵，shape=[batch_size, seq_len]\n",
    "        \"\"\"\n",
    "        # inputs: [batch_size, seq_length, hidden_size]\n",
    "        batch_size, seq_length, hidden_size = inputs.shape\n",
    "        # scores : [batch_size, seq_length, 1]\n",
    "        scores = paddle.matmul(inputs, self.q)\n",
    "        # scores : [batch_size, seq_length]\n",
    "        scores = scores.squeeze(-1)\n",
    "        return scores\n",
    "\n",
    "paddle.seed(2021)\n",
    "inputs = paddle.rand(shape=[1, 3, 3])\n",
    "dotScore = DotProductScore(hidden_size=3)\n",
    "scores = dotScore(inputs)\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上面显示的是点积注意力打分算子的打分输出，点积注意力打分算子的输入是随机初始化的张量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.2.2 注意力分布计算\n",
    "\n",
    "在计算注意力分布的公式中，需要用到Softmax函数计算注意力分布。在实践中，如果采用小批量梯度下降进行优化，需要对同一批次中不同长度的输入序列进行补齐。用$\\mathbf S\\in \\mathbb{R}^{B\\times L}$表示一组样本的注意力打分值，其中$B$是批量大小，$L$是填充补齐后的序列长度，每一行表示一个样本中每个元素的注意力打分值，注意力分布的计算为\n",
    "\n",
    "$$\n",
    "\\Alpha =softmax(\\mathbf S + \\mathbf M) \\in \\mathbb{R}^{B\\times L},\n",
    "$$\n",
    "其中$softmax(\\cdot)$是按行进行归一化，$\\mathbf M\\in \\mathbb{R}^{B\\times L}$是掩码（mask）矩阵，比如[PAD]位置的元素值置为-1e9，其它位置的元素值置为0，$\\mathbf{\\Alpha} \\in \\mathbb{R}^{B\\times L}$是归一化后的**注意力分布**，也称为**注意力权重**。\n",
    "\n",
    "比如第6.4.1.3中的例子：\n",
    "```\n",
    "句子1: This movie was craptacular [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
    "句子2: I got stuck in traffic on the way to the theater\n",
    "```\n",
    "\n",
    "注意力打分函数$\\mathbf S$的的结果为：\n",
    "\n",
    "```\n",
    "句子1: 0.31750774 0.52375913 0.81493020 0.84624285 0.84624285 0.76624285 0.64524285 0.54424285 0.44324285 0.24724285 0.84624285\n",
    "句子2: 0.24595281 0.48540151 1.18520606 0.61489654 1.19498014 0.83661449 0.61444044 0.49837655 0.60015976 0.58790737 0.89794636\n",
    "```\n",
    "掩码矩阵$\\mathbf M$为：\n",
    "\n",
    "```\n",
    "句子1: 0 0 0 0 -1e9 -1e9 -1e9 -1e9 -1e9 -1e9 -1e9\n",
    "句子2: 0 0 0 0 0 0 0 0 0 0 0\n",
    "```\n",
    "公式中的$\\mathbf S + \\mathbf M$则变为：\n",
    "\n",
    "```\n",
    "句子1: 0.31750774 0.52375913 0.81493020 0.84624285 -1e9 -1e9 -1e9 -1e9 -1e9 -1e9 -1e9\n",
    "句子2: 0.24595281 0.48540151 1.18520606 0.61489654 1.19498014 0.83661449 0.61444044 0.49837655 0.60015976 0.58790737 0.89794636\n",
    "```\n",
    "再使用Softmax计算注意力权重，输出为：\n",
    "```\n",
    "句子1: 0.17952277 0.22064464 0.2952211  0.30461147 0. 0. 0. 0. 0. 0. 0.\n",
    "句子2: 0.05510249 0.07001039 0.14095604 0.07968955 0.14234053 0.09947003 0.07965322 0.07092468 0.0785238  0.07756757 0.10576169\n",
    "```\n",
    "可以看到[PAD]部分在填充-1e9之后，对应的Softmax输出变成了0，相当于把[PAD]这些没有特殊意义字符给屏蔽了，然后剩下元素计算注意力分布，这样做就减少了这些没有特殊意义单元对于注意力计算的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "掩码实现使用到了`paddle.where(condition, x, y, name=None)`的API，该OP返回一个根据输入 condition, 选择 x 或 y 的元素组成的多维 Tensor ：\n",
    "\n",
    "$$\n",
    "out_{i}=\\left\\{\n",
    "\\begin{aligned}\n",
    "x_{i}, cond_{i} = True \\\\\n",
    "y_{i}, cond_{i} = False \\\\\n",
    "\\end{aligned}\n",
    "\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "```\n",
    "# arrange: [1,seq_len],比如seq_len=4, arrange变为 [0,1,2,3]\n",
    "arrange = paddle.arange((scores.shape[1]), dtype=paddle.float32).unsqueeze(0)\n",
    "# valid_lens : [batch_size, 1]\n",
    "valid_lens = valid_lens.unsqueeze(1)\n",
    "# 掩码在实现的过程中使用了广播机制。\n",
    "# mask [batch_size, seq_len]\n",
    "mask = arrange < valid_lens\n",
    "y = paddle.full(scores.shape, -1e9, scores.dtype)\n",
    "scores = paddle.where(mask, scores, y)\n",
    "# attn_weights: [batch_size, seq_len]\n",
    "attn_weights = F.softmax(scores, axis=-1)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.2.3  加权平均\n",
    "\n",
    "加权平均就是在使用打分函数计算注意力分布后，用该分布的每个值跟相应的输入的向量相乘得到的结果，公式如下：\n",
    "$$\n",
    "\\mathbf z = \\sum_{n=1}^{N}\\alpha_{n} \\mathbf x_n.\n",
    "$$\n",
    "\n",
    "加权平均的代码如下：\n",
    "\n",
    "```\n",
    "# X: [batch_size, seq_len, hidden_size]\n",
    "# attn_weights: [batch_size, seq_len]\n",
    "# context: [batch_size, 1, hidden_size]\n",
    "context = paddle.matmul(attn_weights.unsqueeze(1), X)\n",
    "# context: [batch_size, hidden_size]\n",
    "context = paddle.squeeze(context, axis=1)\n",
    "```\n",
    "\n",
    "包含加权平均的完整注意力机制的的实现代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F\n",
    "\n",
    "class Attention(nn.Layer):\n",
    "    def __init__(self, hidden_size, use_additive=False):\n",
    "        super(Attention, self).__init__()\n",
    "        self.use_additive = use_additive\n",
    "        # 使用加性模型或者点积模型\n",
    "        if self.use_additive:\n",
    "            self.scores = AdditiveScore(hidden_size)\n",
    "        else:\n",
    "            self.scores = DotProductScore(hidden_size)\n",
    "        self._attention_weights = None\n",
    "\n",
    "    def forward(self, X, valid_lens):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - X：输入矩阵，shape=[batch_size, seq_len, hidden_size]\n",
    "            - valid_lens：长度矩阵，shape=[batch_size]\n",
    "        输出：\n",
    "            - context ：输出矩阵，表示的是注意力的加权平均的结果\n",
    "        \"\"\"\n",
    "        # scores: [batch_size, seq_len]\n",
    "        scores = self.scores(X)\n",
    "        # arrange: [1,seq_len],比如seq_len=4, arrange变为 [0,1,2,3]\n",
    "        arrange = paddle.arange((scores.shape[1]), dtype=paddle.float32).unsqueeze(0)\n",
    "        # valid_lens : [batch_size, 1]\n",
    "        valid_lens = valid_lens.unsqueeze(1)\n",
    "        # mask [batch_size, seq_len]\n",
    "        mask = arrange < valid_lens\n",
    "        y = paddle.full(scores.shape, -1e9, scores.dtype)\n",
    "        scores = paddle.where(mask, scores, y)\n",
    "        # attn_weights: [batch_size, seq_len]\n",
    "        attn_weights = F.softmax(scores, axis=-1)\n",
    "        self._attention_weights = attn_weights\n",
    "        # context: [batch_size, 1, hidden_size]\n",
    "        context = paddle.matmul(attn_weights.unsqueeze(1), X)\n",
    "        # context: [batch_size, hidden_size]\n",
    "        context = paddle.squeeze(context, axis=1)\n",
    "        return context\n",
    "\n",
    "    @property\n",
    "    def attention_weights(self):\n",
    "        return self._attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "使用加性打分函数的代码实现如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "输入向量为 [[[0.04542791 0.85057974 0.33361533]\n",
      "  [0.946391   0.23847368 0.36302885]\n",
      "  [0.76614064 0.37495252 0.33336037]]]\n",
      "注意力的输出为 : [[0.7633098  0.36295402 0.3570773 ]]\n",
      "注意力权重为 : [[0.20322487 0.79677516 0.        ]]\n"
     ]
    }
   ],
   "source": [
    "paddle.seed(2021)\n",
    "X = paddle.rand(shape=[1, 3, 3])\n",
    "valid_lens = paddle.to_tensor([2])\n",
    "print(\"输入向量为 {}\".format(X.numpy()))\n",
    "add_atten = Attention(hidden_size=3, use_additive=True)\n",
    "context = add_atten(X, valid_lens)\n",
    "print(\"注意力的输出为 : {}\".format(context.numpy()))\n",
    "print(\"注意力权重为 : {}\".format(add_atten.attention_weights.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果看，输入向量是一个$3 \\times 3$的矩阵，输入向量采用随机初始化。加性模型的输出是一个$1 \\times 3$的向量，输出的注意力权重就是输入的3个向量的权重，用 $1 \\times 3$的向量表示，即三个输入向量的权重依次为0.20325246，0.7967475和0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "使用点积打分函数的代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "输入向量为 [[[0.04542791 0.85057974 0.33361533]\n",
      "  [0.946391   0.23847368 0.36302885]\n",
      "  [0.76614064 0.37495252 0.33336037]]]\n",
      "注意力的输出为 : [[0.61859894 0.46138203 0.3523724 ]]\n",
      "注意力权重为 : [[0.36400315 0.6359969  0.        ]]\n"
     ]
    }
   ],
   "source": [
    "paddle.seed(2021)\n",
    "X = paddle.rand(shape=[1, 3, 3])\n",
    "valid_lens = paddle.to_tensor([2])\n",
    "print(\"输入向量为 {}\".format(X.numpy()))\n",
    "dot_atten = Attention(hidden_size=3)\n",
    "context = dot_atten(X, valid_lens)\n",
    "print(\"注意力的输出为 : {}\".format(context.numpy()))\n",
    "print(\"注意力权重为 : {}\".format(dot_atten.attention_weights.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果看，输入向量是一个$3 \\times 3$的矩阵，输入向量采用随机初始化。点积模型的输出是一个$1 \\times 3$的向量，输出的注意力权重就是输入的3个向量的权重，用 $1 \\times 3$的向量表示，即三个输入向量的权重依次为0.363956，0.636044和0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.2.4 模型汇总\n",
    "\n",
    "实现了注意力机制后，我们考虑实现整个模型，首先是嵌入层，用于输入的句子中的词语的向量化表示，接着就是双向LSTM来学习句子的上下文特征，随后接入注意力机制来进行特征筛选，最后接入输出层，得到该句子的分类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Model_LSTMAttention(nn.Layer):\n",
    "    def __init__(\n",
    "        self,\n",
    "        hidden_size,\n",
    "        embedding_size,\n",
    "        vocab_size,\n",
    "        n_classes=10,\n",
    "        n_layers=1,\n",
    "        use_additive=False,\n",
    "    ):\n",
    "        super(Model_LSTMAttention, self).__init__()\n",
    "        # 表示LSTM单元的隐藏神经元数量，它也将用来表示hidden和cell向量状态的维度\n",
    "        self.hidden_size = hidden_size\n",
    "        # 表示词向量的维度\n",
    "        self.embedding_size = embedding_size\n",
    "        # 表示词典的的单词数量\n",
    "        self.vocab_size = vocab_size\n",
    "        # 表示文本分类的类别数量\n",
    "        self.n_classes = n_classes\n",
    "        # 表示LSTM的层数\n",
    "        self.n_layers = n_layers\n",
    "        # 定义embedding层\n",
    "        self.embedding = nn.Embedding(\n",
    "            num_embeddings=self.vocab_size, embedding_dim=self.embedding_size\n",
    "        )\n",
    "        # 定义LSTM，它将用来编码网络\n",
    "        self.lstm = nn.LSTM(\n",
    "            input_size=self.embedding_size,\n",
    "            hidden_size=self.hidden_size,\n",
    "            num_layers=self.n_layers,\n",
    "            direction=\"bidirectional\",\n",
    "        )\n",
    "        # lstm的维度输出\n",
    "        output_size = self.hidden_size*2\n",
    "        # 定义Attention层\n",
    "        self.attention = Attention(output_size, use_additive=use_additive)\n",
    "        # 定义分类层，用于将语义向量映射到相应的类别\n",
    "        self.cls_fc = nn.Linear(\n",
    "            in_features=output_size, out_features=self.n_classes\n",
    "        )\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        input_ids, valid_lens = inputs\n",
    "        # 获取训练的batch_size\n",
    "        batch_size = input_ids.shape[0]\n",
    "        # 获取词向量并且进行dropout\n",
    "        embedded_input = self.embedding(input_ids)\n",
    "        # 使用LSTM进行语义编码\n",
    "        last_layers_hiddens, (last_step_hiddens, last_step_cells) = self.lstm(\n",
    "            embedded_input, sequence_length=valid_lens\n",
    "        )\n",
    "        # 使用注意力机制\n",
    "        # 进行Attention, attn_weights: [batch_size, seq_len]\n",
    "        last_layers_hiddens = self.attention(last_layers_hiddens, valid_lens)\n",
    "        # 将其通过分类线性层，获得初步的类别数值\n",
    "        logits = self.cls_fc(last_layers_hiddens)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.1.3 使用加性注意力模型进行实验\n",
    "\n",
    "对于加性注意力模型，我们只需要在双向LSTM的后面加入加性注意力模型，加性注意力模型的输入是双向LSTM的每个时刻的输出，最后接入分类层即可。\n",
    "\n",
    "### 8.1.3.1 模型训练\n",
    "\n",
    "这里使用第4.5.4节中定义的RunnerV3来进行模型训练、评价和预测。\n",
    "使用交叉熵损失函数，并用Adam作为优化器来训练，使用加性注意力模型。模型在训练集上训练2个回合，并保存准确率最高的模型作为最佳模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Train] epoch: 0/2, step: 0/392, loss: 0.69303\n",
      "[Train] epoch: 0/2, step: 10/392, loss: 0.68735\n",
      "[Evaluate]  dev score: 0.49472, dev loss: 0.69193\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.49472\n",
      "[Train] epoch: 0/2, step: 20/392, loss: 0.67718\n",
      "[Evaluate]  dev score: 0.71824, dev loss: 0.67720\n",
      "[Evaluate] best accuracy performence has been updated: 0.49472 --> 0.71824\n",
      "[Train] epoch: 0/2, step: 30/392, loss: 0.63145\n",
      "[Evaluate]  dev score: 0.74440, dev loss: 0.63653\n",
      "[Evaluate] best accuracy performence has been updated: 0.71824 --> 0.74440\n",
      "[Train] epoch: 0/2, step: 40/392, loss: 0.51570\n",
      "[Evaluate]  dev score: 0.74824, dev loss: 0.60993\n",
      "[Evaluate] best accuracy performence has been updated: 0.74440 --> 0.74824\n",
      "[Train] epoch: 0/2, step: 50/392, loss: 0.47718\n",
      "[Evaluate]  dev score: 0.79616, dev loss: 0.47899\n",
      "[Evaluate] best accuracy performence has been updated: 0.74824 --> 0.79616\n",
      "[Train] epoch: 0/2, step: 60/392, loss: 0.34465\n",
      "[Evaluate]  dev score: 0.82424, dev loss: 0.38504\n",
      "[Evaluate] best accuracy performence has been updated: 0.79616 --> 0.82424\n",
      "[Train] epoch: 0/2, step: 70/392, loss: 0.33909\n",
      "[Evaluate]  dev score: 0.84064, dev loss: 0.36226\n",
      "[Evaluate] best accuracy performence has been updated: 0.82424 --> 0.84064\n",
      "[Train] epoch: 0/2, step: 80/392, loss: 0.35559\n",
      "[Evaluate]  dev score: 0.82336, dev loss: 0.39465\n",
      "[Train] epoch: 0/2, step: 90/392, loss: 0.40658\n",
      "[Evaluate]  dev score: 0.85208, dev loss: 0.35200\n",
      "[Evaluate] best accuracy performence has been updated: 0.84064 --> 0.85208\n",
      "[Train] epoch: 0/2, step: 100/392, loss: 0.28806\n",
      "[Evaluate]  dev score: 0.84864, dev loss: 0.34065\n",
      "[Train] epoch: 0/2, step: 110/392, loss: 0.43937\n",
      "[Evaluate]  dev score: 0.85184, dev loss: 0.33583\n",
      "[Train] epoch: 0/2, step: 120/392, loss: 0.29646\n",
      "[Evaluate]  dev score: 0.85832, dev loss: 0.32961\n",
      "[Evaluate] best accuracy performence has been updated: 0.85208 --> 0.85832\n",
      "[Train] epoch: 0/2, step: 130/392, loss: 0.39948\n",
      "[Evaluate]  dev score: 0.84368, dev loss: 0.35792\n",
      "[Train] epoch: 0/2, step: 140/392, loss: 0.34504\n",
      "[Evaluate]  dev score: 0.84912, dev loss: 0.34429\n",
      "[Train] epoch: 0/2, step: 150/392, loss: 0.27905\n",
      "[Evaluate]  dev score: 0.85200, dev loss: 0.34090\n",
      "[Train] epoch: 0/2, step: 160/392, loss: 0.24992\n",
      "[Evaluate]  dev score: 0.85448, dev loss: 0.33347\n",
      "[Train] epoch: 0/2, step: 170/392, loss: 0.28572\n",
      "[Evaluate]  dev score: 0.86672, dev loss: 0.31365\n",
      "[Evaluate] best accuracy performence has been updated: 0.85832 --> 0.86672\n",
      "[Train] epoch: 0/2, step: 180/392, loss: 0.18275\n",
      "[Evaluate]  dev score: 0.86432, dev loss: 0.31826\n",
      "[Train] epoch: 0/2, step: 190/392, loss: 0.23720\n",
      "[Evaluate]  dev score: 0.85568, dev loss: 0.33156\n",
      "[Train] epoch: 1/2, step: 200/392, loss: 0.21320\n",
      "[Evaluate]  dev score: 0.87064, dev loss: 0.30760\n",
      "[Evaluate] best accuracy performence has been updated: 0.86672 --> 0.87064\n",
      "[Train] epoch: 1/2, step: 210/392, loss: 0.25008\n",
      "[Evaluate]  dev score: 0.85440, dev loss: 0.38287\n",
      "[Train] epoch: 1/2, step: 220/392, loss: 0.20038\n",
      "[Evaluate]  dev score: 0.84480, dev loss: 0.37772\n",
      "[Train] epoch: 1/2, step: 230/392, loss: 0.15512\n",
      "[Evaluate]  dev score: 0.84872, dev loss: 0.37544\n",
      "[Train] epoch: 1/2, step: 240/392, loss: 0.14974\n",
      "[Evaluate]  dev score: 0.85912, dev loss: 0.37562\n",
      "[Train] epoch: 1/2, step: 250/392, loss: 0.18064\n",
      "[Evaluate]  dev score: 0.85160, dev loss: 0.37854\n",
      "[Train] epoch: 1/2, step: 260/392, loss: 0.08487\n",
      "[Evaluate]  dev score: 0.84456, dev loss: 0.38121\n",
      "[Train] epoch: 1/2, step: 270/392, loss: 0.15321\n",
      "[Evaluate]  dev score: 0.85560, dev loss: 0.38574\n",
      "[Train] epoch: 1/2, step: 280/392, loss: 0.04542\n",
      "[Evaluate]  dev score: 0.85784, dev loss: 0.40964\n",
      "[Train] epoch: 1/2, step: 290/392, loss: 0.13097\n",
      "[Evaluate]  dev score: 0.85544, dev loss: 0.39555\n",
      "[Train] epoch: 1/2, step: 300/392, loss: 0.04607\n",
      "[Evaluate]  dev score: 0.84952, dev loss: 0.36900\n",
      "[Train] epoch: 1/2, step: 310/392, loss: 0.11698\n",
      "[Evaluate]  dev score: 0.85360, dev loss: 0.39304\n",
      "[Train] epoch: 1/2, step: 320/392, loss: 0.05282\n",
      "[Evaluate]  dev score: 0.85176, dev loss: 0.43203\n",
      "[Train] epoch: 1/2, step: 330/392, loss: 0.15781\n",
      "[Evaluate]  dev score: 0.84728, dev loss: 0.42631\n",
      "[Train] epoch: 1/2, step: 340/392, loss: 0.04877\n",
      "[Evaluate]  dev score: 0.84920, dev loss: 0.35563\n",
      "[Train] epoch: 1/2, step: 350/392, loss: 0.03614\n",
      "[Evaluate]  dev score: 0.82192, dev loss: 0.63461\n",
      "[Train] epoch: 1/2, step: 360/392, loss: 0.05237\n",
      "[Evaluate]  dev score: 0.82256, dev loss: 0.54408\n",
      "[Train] epoch: 1/2, step: 370/392, loss: 0.09265\n",
      "[Evaluate]  dev score: 0.82320, dev loss: 0.58039\n",
      "[Train] epoch: 1/2, step: 380/392, loss: 0.10640\n",
      "[Evaluate]  dev score: 0.84600, dev loss: 0.42932\n",
      "[Train] epoch: 1/2, step: 390/392, loss: 0.11922\n",
      "[Evaluate]  dev score: 0.84624, dev loss: 0.38513\n",
      "[Evaluate]  dev score: 0.81520, dev loss: 0.49123\n",
      "[Train] Training done!\n",
      "训练时间:93.82603907585144\n"
     ]
    }
   ],
   "source": [
    "from paddle.optimizer import Adam\n",
    "from nndl import Accuracy, RunnerV3\n",
    "import time\n",
    "\n",
    "paddle.seed(2021)\n",
    "# 迭代的epoch数\n",
    "epochs = 2\n",
    "# 词汇表的大小\n",
    "vocab_size = len(word2id_dict)\n",
    "# lstm的输出单元的大小\n",
    "hidden_size = 128\n",
    "# embedding的维度\n",
    "embedding_size = 128\n",
    "# 类别数\n",
    "n_classes = 2\n",
    "# lstm的层数\n",
    "n_layers = 1\n",
    "# 学习率\n",
    "learning_rate = 0.001\n",
    "# 定义交叉熵损失\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "# 指定评价指标\n",
    "metric = Accuracy()\n",
    "# 实例化基于LSTM的注意力模型\n",
    "model_atten = Model_LSTMAttention(\n",
    "    hidden_size,\n",
    "    embedding_size,\n",
    "    vocab_size,\n",
    "    n_classes=n_classes,\n",
    "    n_layers=n_layers,\n",
    "    use_additive=True,\n",
    ")\n",
    "# 定义优化器\n",
    "optimizer = Adam(parameters=model_atten.parameters(), learning_rate=learning_rate)\n",
    "# 实例化RunnerV3\n",
    "runner = RunnerV3(model_atten, optimizer, criterion, metric)\n",
    "save_path = \"./checkpoint/model_best.pdparams\"\n",
    "start_time = time.time()\n",
    "# 训练\n",
    "runner.train(\n",
    "    train_loader,\n",
    "    dev_loader,\n",
    "    num_epochs=epochs,\n",
    "    log_steps=10,\n",
    "    eval_steps=10,\n",
    "    save_path=save_path,\n",
    ")\n",
    "end_time = time.time()\n",
    "print(\"训练时间:{}\".format(end_time-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "```\n",
    "训练时间:148.9642140865326\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  if isinstance(obj, collections.Iterator):\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  return list(data) if isinstance(data, collections.MappingView) else data\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAFDCAYAAAB/Z6msAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8XXWZ+PHPc865a9JsTdp0T9qmbVoopbQFCrIolWUAFYUp44yiKKMO6Ci/+bnM4Dg6ruMCojj6c9TRURDEUVRkUZQd2mILpU33fUuTptnvcu49398f9yZNm6VZ7r25SZ/365UXzbnnnvM0Jfc+9/v9Ps9XjDEopZRSSqn8YI12AEoppZRS6gRNzpRSSiml8ogmZ0oppZRSeUSTM6WUUkqpPKLJmVJKKaVUHtHkTCmllFIqj2hyppRSSimVRzQ5U0oppZTKI5qcKaWUUkrlEWe0AxiJ8vJyU1VVNdphKKVy6JVXXmk0xlSMdhwjpa9fSp15Bvv6NaaTs6qqKtatWzfaYSilckhE9o52DJmgr19KnXkG+/ql05pKKaWUUnlEkzOllFJKqTyiyZlSSimlVB7R5EwppZRSKo/kLDkTkatEZKuI7BCRT/Tx+DdEZEP6a5uINOcqNqWUUkqpfJGTak0RsYFvA6uAA8BaEXnEGLO56xxjzEd7nH8HcG4uYlNjT2trK0ePHsV13dEORWWYz+dj0qRJFBUVjXYoSik1anLVSmMFsMMYswtARB4A3gJs7uf8m4F/zVFsagxpbW2lvr6eadOmEQqFEJHRDklliDGGSCTCwYMHATRBU0qdsXI1rTkN2N/j+wPpY72IyCygGngqB3GpMebo0aNMmzaNcDisidk4IyKEw2GmTZvG0aNHRzscpZQaNflYELAa+IUxJtnXgyJym4isE5F1DQ0NOQ5NjTbXdQmFQqMdhsqiUCikU9ZKqTNarpKzg8CMHt9PTx/ry2rg/v4uZIz5njFmmTFmWUXF4HZwMUmPtq+/gNcWG2y8Ko/piNn4pv++SqkzXa6Ss7VAjYhUi4ifVAL2yKknicgCoBR4MZM3dzccofWfnqT5tt9gjDnpscTB47h7dAROKaVUdnmROPEd9b3eh5Q6VU6SM2NMArgdeByoAx40xmwSkc+KyPU9Tl0NPGAy/H+u/7ypFH3uciIPvE7nd0/sZWeMIfb6ASJ/3kJ8R30mb6mUUkqdJLZ2N9Hnt2Pao3itkdEOR+WxnK05M8Y8aoyZZ4yZY4z5fPrYp40xj/Q45zPGmF490DKh8BMXE7hyDs3/+Bjx9YeB9ALkNy7Eriwh+uw24lsOZ+PWSmXdli1bEJERb6T92GOPISI0NjZmKDKlxj6T9Oj8Ux2df6rD3X9sWNdIHG7G3XkU/9nTiW+vp/2R9XiReIYjVeNFPhYEZIVYFqU/uQGrPEzTTQ/htUZTx3024SsW4UwvI/riDmKvHxjlSNV4JCIDflVVVY3o+jU1NRw+fJglS5ZkJmClVLfY2t0k9jSSrG/FO9YBgIkniDy7LTVNGU8M+HyT9Ii+uAOZECSweAa+OZMg4RH7y95chK/GoDMmOQOwKwooe+AdJHcfp/n9J9afiWMRemMtTlU5XnOnrgdQGXf48OHur4cffhiAv/zlL93H1q5d2+fz4vHBfbK2bZvKykocJ1etC5U6M7h7GojXHcK/aBqFf70C/9nTAfBaIyT2HyP67DY6frsBr6P/grP4xgN4LRFCF8xBHBu7OIy/dgru9iMkmzpy9VdRY8gZlZwBBC6eRdHn30TkwU10fOfEG6LYFqFLFxBcWYOI4EVdTdJUxlRWVnZ/lZWVAVBRUdF9rKvyuLKykn/7t3/jtttuo6ysjFWrVgHw1a9+lcWLF1NQUMDUqVP527/925N6gZ06rdn1/S9/+UuuvvpqwuEwc+fO5Wc/+9mQY3/uuee4+OKLCQaDlJWV8a53vYtjx05M7ezdu5e3vvWtTJw4kVAoxNy5c7nnnnu6H//FL37BOeecQzgcprS0lAsvvJDXX3996D9EpXLMGEPs1f1Y5RMInFeVGum2U2+bdvkECm++gNAVi/A64nQ8+mq/68isiYX4F07FmV7WfSywZCb4HKJrdul7jerljPyYXfhPK4k9s5eWjz6O/4Lp+JdOBUCsVAm/F4nT8dsNhC6ZjzO5eDRDVYPQ/I+/x91wJOf39S2ppOTuqzN+3a997Wt84hOf4OWXXyaZTLX7ExHuvvtuqqurOXToEB/96Ef5u7/7Ox5//PEBr/Xxj3+cL3/5y9x7773853/+J7fccgsrV64c9DTq/v37ufLKK7nxxhv57ne/S2NjIx/4wAdYvXo1Tz75JADvf//7sW2bp556iuLiYnbu3NmdvO3bt4/Vq1fzta99jeuvv55IJMIrr7yCbdvD/wEplSMiQsFVizGJZHdSdurjvhllWFedTeeTrxPbsI/QJfN7neebUYZvRtnJzw34CJ47k9jGA5hIHAkHsvb3UGPPGZmciWVR+t9vpeHc79J040NM+svfYxUHTzzu2JhoAnd7vSZnKufe8IY38KlPfeqkY3feeWf3n6urq7nnnntYuXIlx44dY+LEif1e66Mf/Sg33HADAF/4whe49957efrppwednH3zm99k8uTJfP/73++eMv3Rj37EBRdcwJo1a1ixYgV79+7lPe95D+eccw7ASdc+ePAgnudx0003MWXKFAAWLlw4qHsrdSpjDN6xdqyJhVnvh+fubsCZOREJOEhg4LdKu2IC4b86B6sg0B2niODubcRr6sC/eEafyZ1vwRR8NZWIL3MfVuJ1h0gcaSF44VysoC9j11W5dUYmZwB2eQGlD7yDxkt/yPH3PULZgzd2/7KLz8ZXXY67u4Hg+XMy+oujMi8bo1ejacWKFb2O/eEPf+DLX/4yW7Zsobm5Gc/zgNSU4kDJWc8CAb/fT3l5OfX1g28bs2nTJlauXHnSWrYVK1YQDAbZtGkTK1as4GMf+xi33347v/71r7nsssu49tprueiiiwBYvnw5l156KfPnz2fVqlVcdtll3HDDDUyb1ufubUr1y+uIEXl2G8nDzQSWVRNIr/3KBnd3A5E/byGwvJrAWYO7j10cBlKFAp1/3Ix/0TSiL+1Egj7858zs8zliWWClCga85k7siYUji3vnUaIv7QSgo7GN8JsWYZcVjOiaXYwxqcKIIy14bVHsigk4U0pwZpRhl4Qzcg91whm35qynwEUzKfriFUR/sZmOb6856TFfzWRIeNqgVuVcQcHJL6Y7duzg2muvZf78+fz85z9n3bp1PPTQQ8DpCwb8fv9J34tId2KXKX//93/P7t27ufXWW9m3bx+rVq3ife97HwCO4/DUU0/xxBNPcO655/LAAw9QU1PTPSWq1GBF1+wi2dCKVVZAbMNevLZoVu6TbIkQeX47dsUE/AunDvn5JuFhoi6RP27GdMYJrpzbvWSmP9EXd9Dx2Ea86PC3LUu2dBJ5bht2ZTHhaxaDCCSH/7vudcSIvLSjO9kTERL1LanBi1kTMe1RYut246Z7hJqER3zrYcwI7qlOOKOTM4DCOy8k8Fc1tHzsceLrTuwoZU8qwioK4W7X5rRqdL388su4rsvdd9/NypUrmT9/PkeO5GaN3aJFi3jhhRdIJE60ClizZg3RaJSzzjqr+9j06dN53/vex09/+lPuu+8+fvCDHxCLparXRIQLLriAf/mXf+H5559nxYoV/OhHP8pJ/GpsM/EEXmfqA0jw/DkUXL+U8JsWAULstf2ZvZcxxF7bT+TJ10GE0GULUiNbQ2SF/YSvXow9tQT/OTNwKopO+xz/omngJoi9smfYxQFWUYjghXMJv3EhzuRiCm84D7tiApDaCWco1zXxBJ1PvI67rf6kXmwF1y6h4OrFhC6eR+ENy1LVq+kENtnYRvSFHXQ8sp7E0dZh/R3UCWd8cpZaf/Y27MrCVP+z5lS1jYgQOK8K/0KdflGja968eXiexze+8Q12797Nww8/zBe/+MWc3PsjH/kI9fX1vO9972PTpk08/fTTvOc97+GKK65g+fLlAHzgAx/gscceY+fOnbz++uv86le/Ys6cOQQCAf785z/zhS98gTVr1rBv3z6eeOIJNm/enLfrzkTkKhHZKiI7RKRXQ2wRmSkifxKR9SLymohckz5eJSIREdmQ/vrP3Ec/viTqW2j/9Xoiz27FGIMV9mMXh7AKA4SvPIvg+XNGdH2vPUp825Hu3pYigrvzKDgW4ctrsQqDp7lC/6ygj4Irzya4tGpQ59ulBfgXTcfddoT4hn1DupfXHiV5vAMRwT+vsnt9XFdimahvofOJ14k8VYdxk6e9nvEMkae34rV0Er5iEeHLa7sfO3WdnxUOYKULGezJRYSuWIRxk3T+7lWiL+3EuL37v5mkR3x7PfFtR7rv1/HbDUSe25ZaL3e0FZM4fZzj3Rm75qwne2KY0p/fSOMlP+T4e39N2cN/narCqSof7dCUYvny5Xz961/na1/7Gp/+9Kc5//zz+cY3vsF1112X9XtPnz6dxx9/nI9//OOcd955hMNhrr32Wr7xjW90n5NMJrnjjjs4cOAA4XCYlStX8pvf/AaA0tJSnnnmGe655x6am5uZMmUKt956Kx//+MezHvtQiYgNfBtYBRwA1orII8aYzT1O+xdS2899R0QWAo8CVenHdhpjtAtwH7yom1pYL0LicDNWcRgr7O91nom5uHsacXc1kDzSghQGCSyZ1SspcCalRqOMmwRjEP/g3sqSzZ0kdjfg7juGl+4vZpWE8S+alqrMvG4J4ozOGuPAsipM1CW2YR8S8uFfcPopVRNz6XxyE8ZNUvj2ZX0WHdiTigismE1s7a5UF4I31navj+tLbP1eEgeaCF44B2dqyaDj76pcdSqXEn1lD/G6QySPd1Bw9WIg9f+Au+Uw8S2HMBEXe0oJ/nmV4CYQv0Nif9OJmSoRghfMwb9gyqDvP97IWO6vsmzZMjPS7Wp6avvaC7T+nycovucqCj98AZCad3d3NaR+eU+zbkBlX11dHbW1tac/UY1pA/07i8grxphlmb6niFwIfMYYc2X6+08CGGO+2OOc7wK7jDFfTp//NWPMShGpAn5rjDmr95X7lunXr3zW+cTrGDdJ+OrFtD/4Mibq4kwtxVczGXtKCeJYiGMTrztE9KWdWEUhfLMr8J81DfH1nXiZRJL2X/0FZ0oJoYtq+r13or4Vu7wQsS2i63YT33gAe1IRzsyJODNKsYrDWa/8HCzjGWLrduOvnYo1YeCRO5P06HzidZJHWwm/+SycKQMnUomDx4k8vQXjGUIXz+t38CHZ1E5i3zECS2YN++8BqZ87xuBUFhPfcpjoml2Q9LCnlRJYNA17aslJP3djDKYjTvJYG4lDzfjnT8EuK8Bri2ASHnZp34UNXtTFa+7ALi9CnPyfDBzs65eOnPVQ+LELiT+9h5b/80Sq/9mK6SSPthJbtxu7rABnWuloh6iUyp5pQM+FTAeA80855zPAEyJyB1AAXNHjsWoRWQ+0Av9ijHn21BuIyG3AbQAzZ/ZdwTfemHiCxOFm/AunIpYQvnox7o6juDvqSfx5CwDBi2rwz6vEN7sCu6IIa2LBaRMmcWx8MycS33QQ39xJvdoeea0Romt2kdjflEpeppXiXzgN/6JpWKHeo3b5QCwhuGI2kEpWkkfbcCb3XrPm7m4g9tp+vKYOQpfOP21iBuBMK6Xg+qVE/lzXZ6GA1xlPTR2XFWKXjaxqFDgpbqskjG/OJPyLpmKX9J1kiQhSGMAqDOCbdSJxjG3Yh7vjKM6sifjPmg6eAUtwJhVh4gna738JgMDSWQT6qYodi/I/zcwhEaH0R2/FnjqBpr/+Bd7xSHefm/j23Dc5VUrlnZuBHxljpgPXAD8REQs4DMw0xpwLfAz4mYj0elc1xnzPGLPMGLOsa1eI8S6xvwk8g5N+w7WLwwTPq6LwxhWErzwL/5KZ3S0kJOBLjXINciQrcO4spDBA9IUd3VWCxk0SfWUP7f/7ConDLQSWVWOnp0GtsD9vE7NTxTcfovPRV3F3pXYC8Tpi3Yv6kw1tkPQIXTof3+xJg76mVRggfM05qb09ITW92x7Da4vQ8atXMl5k0cWpLCZ0UU2/idlAAstn4z9nBonDzXT+7lU6f/8a8VdTcYrfSfVzm1iIuyt7nRW8SJzE0VYSR5pJHDyOu/8YJjbwfqojpSNnp7DKwpT9/EYa3vADjr/nV5T972qc2RW4245gYi4S0KZ+So1TB4EZPb6fnj7W063AVQDGmBdFJAiUG2OOArH08VdEZCcwDzgz5i0H4O5tREL+7srBLmIJztRSnKnDn5EQn03wgrlE/rCJ+MYDBJbM7J7q882ZRGBZdZ9r28YC//xKEnsbiTyzFXfHURKHjqdGAKeWpkaJllcPazq2a3mOcZNEn9+eWrMX8GEM3Ql0PrGCPoJLqwicNR13T/r/pfITI3v+BVPAGKIv7SR5vKPf6c/h6ipYMO0n751acO2SXv9PZ5KOnPXBf/50ir+yiuivt9Jxz0v4ayohabKamSulRt1aoEZEqkXED6wGHjnlnH3AmwBEpBYIAg0iUpEuKEBEZgM1wK6cRZ6nTCJJ4uBxnFkTs7auyzejDKeqPFXlZwyBJTMJX7OY0CXzx2xiBqlp2/CqRdjlE0g0tOFfNB2rKNz92Eh/nuKzCV+zGAn58dqihC+vxS4OZSL0rBC/k5r6nlHWa/SzK6lM7G/K+H1NzMUqDhNYVkX4yrMJX7OYgmuXYJWE6Xh8I51/2JTxe4KOnPWr4CMXEHt6Ly3/9CT+C2dglRfidcRO/0Sl1JhkjEmIyO3A44AN/MAYs0lEPgusM8Y8AtwJ/D8R+ShggFuMMUZELgE+KyIu4AEfMMZk/p1irLGE0OW13dsaZUvo4nngWIjIuFobLD6H8DXnAGZYPddOxy4OU3DduZhI/LQFCPnMCvspeOtSrCzsVGCF/BS8uZ86H89gktmZ3tTkrB8iQukP3sLRpd+l6aaHqHjlNuzyzA6XKqXyizHmUVLtMXoe+3SPP28GLurjeQ8DD2c9wDyRbOogsbcR/5KZA47giGXhm17W7+OZMp632EtNQ2avmlQcCxnDiVmXTE9nQmqtGUmDVdjPhwvbgnh2kjOd1hyAVRqi7MEbSR5uo/m9v06V+mbpH0IppcYC4xk6H99IfOfR1Nhhf+clPaLr9+K1RnIXnDpjmfS6s66mwpkQ33yI9ofX9rutlliSte2qNDk7Df/yaRR/9c1Ef7ONtu88T/vD67R7sVLqjOXuqMdEXYLnVQ3Y+zF5pIX4hn0kmztzGJ06U4kIyZZO3K1Hhr0FVk/G83C3H8GZVooV7KcQ0LZGtH/pQDQ5G4SCO84neEMtnfe+gom6xDOYmSul1Fhh3CSxv+zBrpiAU1VOfOvh1AhaH9w9jeBYI6rGVGoofFXleK2R7t0fRiKxrwkTcfHN73+XAmdqSdYqXDU5GwQRofS/roeYIbGpidhrB/DatThAjT233HILV1xxxelPTDOJpE7lq26xjfsxEZdAulGqu6uB2JpdvfZsNJ4hse8YzvSyMdG1XY0PzqxykFST3pGKbz2MFAQGLDDxz59C8LyqEd+rL/pbM0hWSYgJ/3oZ0R/VgWeIvrJ7tENSY8wtt9yS6oItgs/no7y8nIsvvpivfOUrdHSM/JNeNnjNnXitEYyXnaF7NbZYE4L4aqfgTCpCRAicl9oPMr7p5HZwyaOtmKh7Uqd3pbLNCvqwp5bi7m4c0dSmF4mTPNKS2kj+NNs2ZmsLTE3OhsC3qAJzLIrlD5I4cDxVyaHUELzhDW/g8OHD7N27lz/96U+8853v5Fvf+hZLly6lvr5+tMPrX5bWVaixxV9TSeiCud3fO+k9KmOvHzhp0bTXFkECDs4MndJUueWvmZzasD0x/NcsK+Sn8MYV+GsH3ng9um43bf/zwrDvM2AMWbnqOOUsSH0K9PZ0Uvj288bMNiAqf/j9fiorK5k6dSpnn302H/zgB3nxxRdpaGjgE5/4xEnn3nvvvSxYsIBgMEhNTQ2f//znSSRSU4z//M//zPz583td/4Mf/CAXX3zxoOMxxvDVr36V2bNn4/f7mTNnDnffffdJ5zzy+9+xdNkywuEwJSUlrFixgvXr1wPgui4f+9jHmD59OoFAgClTprB69eqh/lhUnks2thPfchjj9R4lCJw3CxJJ4j22/vHXVFK4+vx+Ny1XKlt81RWELqoZcXsVK+w//Y5AIpD0sjJ6pr85Q2AVBrBnFpPY3IAV9GOMwWuN5nVX5TNFx+9f63XMV1WOv3YqJpGk88neXZx9cyfjr5mMF3WJ/Kmu1+P++VPwza7Aa48ReXZrr8cLrl6ckdinTZvGO9/5Tn784x/zX//1X1iWxWc+8xl++MMfcvfdd7NkyRLq6ur4wAc+QDQa5XOf+xzvfve7+cIXvsDLL7/M+een9uaOxWL8/Oc/50tf+tKg733fffdx1113cc8993D55Zfzxz/+kX/8x39kwoQJvPe97+VIfT1/feu7+Nxd/8pNf/s3RKNR1q9fj+OkXjruvfdeHnzwQf7nf/6H2bNnU19fz/PPP5+Rn4vKD8YYomt24rVE8M2uAP/Jbxt2SQGBJbOw0lvqGM8glmSlaapSg2GMwTvWgTWxYMg7Kbi7GohvO0zo0gWnHYAR20q1kzEmlahlkCZnQ+TUlpOoawQg+tJOEnsaKbxhGRLQH6UavkWLFtHa2kpjYyOFhYV85Stf4Ze//CVXXXUVANXV1fz7v/87H/7wh/nc5z7HvHnzOP/88/nxj3/cnZz95je/IRKJcNNNNw36vl/60pe44447uO222wCoqalh69atfP7zn+e973kPh+uP4Lou73jLDVRXVwNQW1vb/fy9e/cyb948Lr30UkSEmTNnsnz58kz9WFQeSOw7RrK+leCFcxF/369zgSUzu/8cW7+XxKHjFPzVktOu11EqGxK7G4k8vYXwtefgVBQN6bnxrYfx2mNIf+0zerLTH0CSJuPzkJpRDJFTW0Hns69gPA//vErcLYeJvbqPYLp6SY2OgUaxxLEHfNwK+gZ+vDCQsVGy/nQNi4sImzZtIhKJ8Pa3v/2kT33JZJJoNEpDQwMVFRW8+93v5q677uLuu+/G5/Px4x//mOuvv56SkpJB3bO1tZUDBw5wySWXnHT80ksv5Z577qEzEuGc85fx5itWcc7K5axatYrLLruMG264gRkzUvuDv+c972HVqlXMnTuXVatWsWrVKq677jr8fp3yHw9M0iO2djdWSRjfvMqBz40niG08gLv9CFZJgSZmatQ400rBEhK7G4eUnCWbO0keaSFwXtWgRtzETm8in/QyvkuFjjsPkW9hBabTJbmvBXtiIb55lcQ3H9JGi2pENm3aRHFxMRMnTsRLV0Y+9NBDbNiwoftr48aNbN++nbKy1HY4q1evpq2tjd/97nc0NDTw2GOP8e53vztjMYkIvoIgjz3xOE899RTLly/n4YcfZt68efz2t78FYMmSJezevZuvfvWr+P1+PvKRj7BkyRJaW1szFocaPe62I3htUQLLq09fteYZ4nWHUr2hZk3MUYRK9SYBB2daKe7uhkGvBzNJL1V1bAm+msmDeo41sRD/omlZ+SCiydkQObWpooCuqc3A0lngs4it1dYaangOHjzIT3/6U2644QYsy2LRokUEg0F27drF3Llze33ZduoTWmlpKddddx0/+clPuP/++ykrK+PKK68c9H2LioqYPn06zzzzzEnHn376aaqrqwkFgxg3AcawfMlSPvnxT/DMM89w6aWX8sMf/rD7/MLCQt72trfxzW9+k3Xr1lFXV8fTTz+dmR+OGlVSmG6dMYjNxK2gj8DiGWALjiZnapT5qiswnXHcusNAamQ38sxWOp98nY7fbqD9l+tou/+l7u2eTEcMd9sRnFnlgy72cyYXE1wxu9/p/pHQac0hcmorAHDrGgheXYMV8uOfN6U7Qx/q4kN1ZonH4xw5cgTP8zh27BjPPfccX/ziF5k0aRJf/OIXgVSy86lPfYpPfepTiAhXXHEFiUSCjRs3sn79er785S93X+9d73oXN954I3V1dbzzne/sTtwG65Of/CR33nknNTU1XHbZZTz11FN85zvf4dvf/jYm4fH8H/7Mn9Y8zxUXvoGp1TPZuX8vr732GrfeeisA//Ef/8HUqVNZsmQJ4XCY+++/H9u2mTdvXuZ+aGrU+GaU4Zsx+I3L/WdPx1czWSvZ1ahzZpSB3z6xt6slJOpbkYCDBBysggAScLo3TJdwgNAba3EqB7csBFKjxSSTYNsZHz3LWXImIlcB9wA28H1jTK+SMhG5CfgMqfqHV40xf5Or+AbLnhjGmlRAYvOJDsSBZVUEl1ePYlRqrHj22WeZMmUKtm1TXFxMbW0tt99+O//wD/9AQUFB93l33XUXU6ZM4Vvf+hZ33nknoVCIefPmccstt5x0vauvvpri4mLq6uq4//77hxzPBz/4QTo6OvjCF77Ahz70IWbMmMGXvvQlbr31VryYS3FRES+9/BL3fec+jjc3U1lZyTvf+U7uuusuIDX69vWvf53t27fjeR61tbU8/PDDfbb5UGNLsqkdCfiwCgKDfo6IIJqYqTwgfocJN6448b1jM+HG/ouVxLGG3DQ5caCJyB83U3DdEuzyCcOOtc94stXd9qSbiNjANmAVcABYC9xsjNnc45wa4EHgjcaY4yIyyRjT96ZtacuWLTPr1q3LYuR9a7jsh+B6VDx/a87vfaarq6s7qVpQZY8XiWM6YlilBXjtUTBgl4Rzcu+B/p1F5BVjzLKcBJJFo/X6NVgdj76KiScpfOvS0Q5FqbyUOHiczideJ3zNYpzJxYN6zmBfv3K15mwFsMMYs8sYEwceAN5yyjnvB75tjDkOcLrEbDT5aitwN59YaOhF4nT+cTOJg8dHOTKlMqjrc5tIqp9PlpotqvzjdcZI1rfiq9K1Y0r1K12tSTLzr4u5Ss6mAft7fH8gfaynecA8EXleRF5KT4P2IiK3icg6EVnX0DDyzU2Hw1lYgWmO4tW3p2Jy7FQvoGPtoxKPUlnRlYgJqX4+xkAXUCPFAAAgAElEQVQfHeLV+JPYewwAp0r3xlSqP+L3YU0sBCfzqVQ+FQQ4QA1wGTAdeEZEzjbGNPc8yRjzPeB7kJoWyHWQcHLFpl05AfHZSMDB64iNRjhKZYUEHPDZqSKXgIM4NmjvqjOCu6cRqziEXVJw+pOVOkPZZQUUXn9uVq6dq5Gzg8CMHt9PTx/r6QDwiDHGNcbsJrVGrSZH8Q2Jr6tis0dRgBQEMJqcqXFEHBsrXSIulpX6EKLVyOOeiSVI1rfqqJlSoyhXydlaoEZEqkXED6wGHjnlnF+RGjVDRMpJTXPuylF8Q2JNnYAUBUjUnUjOrIIAXrsmZ7mg655yw7iJVJ+zNC/m4sXc7N9X/31HlQQcCm9agb926miHotQZKyfJmTEmAdwOPA7UAQ8aYzaJyGdF5Pr0aY8Dx0RkM/An4J+MMcdyEd9QichJe2wCWGUFSGgQe3GpEfH5fEQikdEO44zgdcbxOuLd35toAhOJD/CMzIhEIvh8+rs0mqywX3uVKTWKcrbmzBjzKPDoKcc+3ePPBvhY+ivv+WoriD62o/v74NKq0QvmDDJp0iQOHjzItGnTCIVCOs2WTQbo8fMVx8JEkllrtmyMIRKJcPDgQSZPHtz2KSqzvKhL9NmtBJbMwq7IbN8mpdTg5VNBwJjiLKzA+9EGvOYIVklotMM5YxQVpTaxPXToEK6b/Sm2M5nXGQPLwgqmRrGMm8TEXKQ+0G83bJP0wJJhJ28+n4/Jkyd3/zur3ErsO0biwHEC51aNdihKndE0ORumropNt66RwIUzSLZ0Enl6K8FlVThTT78PnRq+oqIiffPOgbYHXsKZMZHQuam6nGRDGx2/3UDojTP67KTtdcZp//nLODPKCF+xMNfhZsTpdjIRkZnAfwMl6XM+kZ4VQEQ+CdwKJIEPG2Mez2XsmZDY04gUBrEmapWmUqNJNz4fpq6Kza5tnMRn4x1rP7GPl1JjnHGTiO/EXp1WencAr6Wf/8fTg2VjtaVMeieTbwNXAwuBm0Xk1CzzX0itmT2XVGHTfennLkx/vwi4Crgvfb0xw8RcEoeb8VWV63IBpUaZjpwNk11VAkGnu2JTQn6wRCs21bgRvvLs7ilNSH0AKVx9PhLse7G+FfJjVxaP5Ua13TuZAIhI104mm3ucY4CuYdti4FD6z28BHjDGxIDdIrIjfb0XcxF4Jrj7m8Az+LSFhlKjTpOzYRLbwjd/Im66YlNEkHBgzI4aKHUqZ1LvqeP+KvgSR1og6WGF/SSOtmY7tGzpayeT80855zPAEyJyB1AAXNHjuS+d8txTd0HJa+LYONPLsMoLRzsUpc54Oq05Ak5tRfe0JoBVqI1o1fhg4gni24/gtUVPOp443Ezk6S2phf89xNbtJvrSTuxJRdh9JHXjyM3Aj4wx04FrgJ+IyKBfR/Nh+7n++KrKCa9apFOaSuUBTc5GwFlYQXJvM15nqveTXVncvS5HqbHMa48RfW57r/1iTWccd1fDSWsrk41tJBva8C2Ygr92KuFLF+Q63EwZzE4mtwIPAhhjXgSCQPkgn4sx5nvGmGXGmGUVFRUZDH1kvPYYxk2OdhhKqTRNzkbAV1sOBhJbU71yg+fOIrQyL3ecUmpIunYG6FkQAGCVposCjnd2H4vXHQLHwl8z5nuTDWYnk33AmwBEpJZUctaQPm+1iAREpJrU1nNrchb5CMX+sof2X72iuzMolSc0ORsB55SKTaXGCxNPj6L4T0nOisIgkGzuAMCLxnF3N+CbOxnxOySPtdP2sxdJHDye65BHbJA7mdwJvF9EXgXuB24xKZtIjahtBh4D/sEYM2aGopJN7djFYZ3SVCpPaEHACDg1ZWBLd8Vm4mgrkT9uJvTGWpzJxaMcnVIj0D1ydvJLhDgWVlGoe+TMa4kgAV/3PozidzCxxJgtjBnETiabgYv6ee7ngc9nNcAsMIkkXnMnzoyJox2KUipNR85GQPwOztyyExWbfgcTdTHaTkONcV0jZ6dOawJYEwshPf3lTC6m8MYV2Om1lpKu5jSd2d+DU2WGd7wTDNgTtUpTqXyhI2cj5Cw8UbFpFQSAsduEU6kuvuoK7EkTupOtnkKXzEdE8DpjSNCHWCc+44ljIQEHLwcbpKvMSDalij5s3RVAqbyhI2cj5NRWkNjRhIknEJ+demNqj57+iUrlMQk42GWFfe6h2bUuKfL0Vjof29j78bBfW8qMIXZlMcEL5iCFwdEORSmVpiNnI+SrLYeER2JHE76Fk5AC7XWmxr7EoeN47TH88yp7PeZF4nT87yuYWILAsupej/uqK8AZUzsXndHs4jB2sbYAUiqf6MjZCDkL0xWb6XVnvuqK1BY2So1h7s4GYhv29fmYBFKL/gH883q3zwicM5PAojHVHP+MZTyDu6dRp6GVyjOanI2QMz+1D52brtgMLJ5B4OwZAz1Fqbxn3ESfxQAAYlk4MyfiWzAFCfS9z6ZJeNozawzwWjqJ/KluTLY+UWo802nNEbIK/Nizik/qdWYSHljS53odpcYC4yYRf/8vD+E3Lez3sfi2I0Sf307hX5+PhPvei1Plh2RTql+dVmoqlV905CwDnIUV3dOa7t5jtP3kebzmztM8S6n8ZeJJ6Gfk7HS6RtNMp669zHfesXawLSxdc6ZUXtHkLAN8tRW4WxoxSQ8Jp96YvA6t2FRj2ADTmqdjpUfLPO11lveSx9qxSsM6yq9UntFpzQxwasshmiC5txmrMjU9oI1o1VgWvmYxDHPJWNdUpjaizW/GGJJNHfiqy0c7FKXUKTQ5y4CeFZuB6lKwRBvRqjHNCg5/rVhX41qtAMx/hdctAR00Uyrv6LRmBvjSG6C7dQ2ICFIQ0ORMjVnGM0T/sofE0dZhPV8swX/ODJxJRRmOTGWSiGAVhbAmhEY7FKXUKXTkLAOs0hDW5ILuis3AwmlIsO8WA0rlPTdJ/NX9SMA37AQruLQqszGpjHP3NmIiLv4FU0Y7FKXUKTQ5yxBnYUX3Buj+hVNHORqlhs+4qQazwy0IADCJJCaawCoMZCoslWHutnq89qgmZ0rlIZ3WzBBfbQWJugaMMZikh9cawXjahFONPSaeBED8w0/Oomt20fGb9ZkKSWVBsqld+5splac0OcsQp7Yc0xLDO9yGu/Mo7Q+v0z5Pakw6MXI2/IF1K+zHRF1M0stUWCqDvEgc0xnH0uRMqbykyVmG+BZ2FQU0YhWkpnI8baehxiDjpkbOhtuEFkDCqd8BoxWbeck71g6AXVYwypEopfqiyVmGOLVd7TQakHRyZrRiU41BzrRSJvzthdjlwx9V6W6nob3O8pLXFgUBu0xHzpTKR1oQkCFWZSFSHCCxuaF7EbTXrrsEqLFHRGAEU5pwYpcAbUSbn/y1U/HNnTyiog+lVPbkbORMRK4Ska0iskNEPtHH47eISIOIbEh/vS9XsWWCiOBLV2yKYyMBR3udqTEpcfA40bW7RrRezJoQJLBiNlapTpvlK03MlMpfOUnORMQGvg1cDSwEbhaRhX2c+nNjzJL01/dzEVsmOemKTYDAitn4Zk8a5YiUGrpEfQvx1w/CCPZbFL9DYNE07GJtcJpvTDxB5xOvkzjcPNqhKKX6kauRsxXADmPMLmNMHHgAeEuO7p0zTm05Xn0HXlMn/rmTcSqLRzskpYbOTYLPTk1vjoDXFiHZ3JmhoFSmJJs6SBw8jkloJa1S+SpXydk0YH+P7w+kj53q7SLymoj8QkRm9HUhEblNRNaJyLqGhoZsxDpsPSs2vWicxJFmjNFeZ2psMW4yI1Nekae3En15ZwYiUpmU7KrU1DYaSuWtfKrW/A1QZYxZDDwJ/HdfJxljvmeMWWaMWVZRUZHTAE+nZ8Wmu+Monb/fCPHEKEel1NCYeALxj7xWSMJ+rVjOQ15TOxLydRdtKKXyT66Ss4NAz5Gw6elj3Ywxx4wxXa/k3wfOy1FsGWPPKkZCDm7Pik19c1JjTdJkZOTMCgfwxlifs0EULn2jR9HSNhFp7vFYssdjj+Q28sFLHuvA0hYaSuW1XLXSWAvUiEg1qaRsNfA3PU8QkSnGmMPpb68H6nIUW8aIZeEsKCdR14hVEARSjWi1l5AaS8KrFmVkOl7CfognMzZNmm09CpdWkVp6sVZEHjHGbO46xxjz0R7n3wGc2+MSEWPMklzFOxzGGCTow5k8vA3tlVK5kZORM2NMArgdeJxU0vWgMWaTiHxWRK5Pn/ZhEdkkIq8CHwZuyUVsmebUVpDYrI1o1dg20mIASCdnDLxLQGzDPjp+/9qI75UhQy1cuhm4PyeRZYiIUHDV2QTOmTnaoSilBpCzJrTGmEeBR0859ukef/4k8MlcxZMtTm05kZ9tTPWIskS3cFJjTuSlHTgVRfjmjKwVjFNZTOjyBUjQ1+85XkcMryVvKjr7Klw6v68TRWQWUA081eNwUETWAQngS8aYX2UrUKXU+KY7BGRYV8VmclsToctrsbTPkxpj3O1HEcsacXJmFQaxCoMDn2QMZGCUbhSsBn5hjEn2ODbLGHNQRGYDT4nIRmPMSeWqInIbcBvAzJm5G70ynkdi3zGia/dgFQYouHpxzu6tlBq6fKrWHBe6KzY3N+CbORG7ODzKESk1eMYzkMjMGjFjDIlDzSSbO/o/yTMjanabYactXOphNadMaRpjDqb/uwv4MyevR+s6J6fV5l5njNj6vbQ/tJbIn7YABv/8yqzfVyk1MpqcZZgztwwcC7euAa81Qnzn0dEOSanBc9MDQRlopQHQ+cdNuNvq+33cGJOR9W0Z0l24JCJ+UglYr6pLEVkAlAIv9jhWKiKB9J/LgYuAzac+N9fcnUeJbdiHVVpA6IqFFL59ue5cotQYoNOaGSY+G6emjERdI+7+JmJrduFMK8EKak8hlf+Mm+rLl4mRMxHBCvnxBtj83C4JI3kycmaMSYhIV+GSDfygq3AJWGeM6UrUVgMPmJNLWmuB74qIR+pD75d6VnnmmheJY4X8+OdV4ptVjlWkyyuUGks0OcsCp7aCxOtHsUtSU5pecwSrUpMzlf9M0kMCTkaa0EK6EW1n/0Ux+VY1eLrCpfT3n+njeS8AZ2c1uEHyWiO0P7yO0CXz8c2ZhAT6L8hQSuUnndbMAl9tOYmdTd2tBDzdX1CNEXZxmAl/cyG+qvKMXE/CAcwAI2cq89xdqW3tbN3bV6kxS5OzLHAWVkDSkDzYDo6tyZk6Y1lhP14k3m9T28hz2+j8w6YcRzV+GWNwdx3FrizGSvdaVEqNPTqtmQVdFZvJukaskjBJTc7UGJE40kx88yGCF8zBCo/8zd03fwrOAKNwXmccE9P9ZzPFa+rAa4kQXDRttENRSo2AjpxlgTN/Igi4dQ2ELplH6LL5ox2SUoPitURI7D0GI9+9CQC7OIQzqaj/ikxj8qYgYDxwdx0FSzI2La2UGh06cpYFVtiPXVVCoq5R+5ypMcWkW2lkai9ME3Nx9zfhTC7GmtBHQ1rPgOZmGeM/azr25GItAlBqjNORsyxxaitwNzfgdcaJbdinU5tqTOhKzshYcpYg+uw2EvUt/ZwwZncIyEtWyI9v5sTRDkMpNUKanGWJb2EFia2NmHiC2Pq9JPt7c1Iqn8QT4LMz1hhWQunNz/up2LQnF2tVYYbENh3E1abXSo0LOq2ZJU5tOcSSeI0RsC2t2FRjg8/GyuBUvPhs8Nv9JmfB86oydq8zmUl6xDbswzejbMR7oiqlRp8mZ1nSvcemVmyqMSS4tAqWZvaaqV0C+m9Eq0YucfA4xBM4s7O/X6dSKvt0WjNLfLWpaqlEXQN2SVhHztQZSwr6b0Tb8eirRJ7dmuOIxh9311Ek4OBMLRntUJRSGaDJWZZYJSGsKYXdI2cmljix2FqpPBV5fjvRV/Zk9JqhlTWE3riwz8dMLIFJeBm935nGuAkS+5pwqisQS1/SlRoPdFozi3zpik3/wqn4z5qu/ZxU3kvWt2CVFmT0mn220OiirTRGzOuIY5WE8emUplLjhn7MyiJnYQWJugawLU3M1Jhg3GTGepx1STZ3Ent1X587ARhjMlYZeqayS8IUXn8uzmStelVqvNDkLIuc2nJMWxzvYCvRl3cS33J4tENSakDZSM685k5if9mL1x7t44YG9IPLsBk3iXF1+yulxhtNzrLIl67YdOsaSRxpIbH/2ChHpM4UyaZ2Is9sxd3TgEkObk2X8Qy4SfBndrWDhFO9zrw+igJ8MydiTyrK6P3OJO6OetrufxmvQ6thlRpPdM1ZFjkLu9ppNGCdU0LyaOsoR6TOFLFX95PY05hqSup38M2uwL9gCvZA68k8D6u8EKtwgDViw2CFuxrR9k4ggufPyei9zjSJA01YhQGsgpFvUq+Uyh+anGWRNakAKQ3ibm4gfMlUErsasjJtpNSpQivnkqydCkmP+I563O312BMLB0zOxLEpvO7cjMci4QD4HZJHWmD+lIxf/0xmYglEEzOlxh1NzrJIRPDVVnS30wDwWjqxyyeMcmRqvJOADye9LZIzrRQTT4AleJ0xEvub8M2elLMPCWIJvqpykkdaehUAtN3/Er45kwiumJ2TWMYbk/Cw0ltkKaXGD11zlmVdFZtWaRgpDKbeJJXKEmMMnU9txt3fdNJx8TuIY+M1dRB9YQfJxrZez00ea6f9138h0ZD56ffgsmoK3nZer8pM7XE2QokkOPoyrtR4o7/VWebUluM1dIJrmHDjcpyppaMdkhrHkoebSew9ltrAvA92RWrUNtnQOzkzURevqQOykC9JwEEswRhzyk0NaCuNYfOfNR1ftfY3U2q80eQsy3w99thUKtviW4+ktvGZVd7n4xLwYRWH+ixO6RrVFX92pjvdfcdo/8U6TMw9cdDTVhoj4V8wBd/MiaMdhlIqwzQ5y7KeFZux1/bT8dhroxyRGq+8aJzEvmP45kxCBpjqsismkGxo6zWK1bW9mPiysxTVCvkx7VHcvT1ayhjdIWAkki2dfTb3VUqNbZqcZZk9owgJ+3A3N0DSI3m4BZPQPTZVaqSq1zTfCLjbj4Jn8M2vHPA8u6IIE3Uxp/TGyvbImVVeiDUhiLurIXU/Y/AtmIJdoX3OhsMkPTp++QrxLYdGOxSlVIZptWaWiWXhLCg/pWIzgj2xcJQjU6Mp2dJJxyPrcSqLCV1eizgjT4isCUF88yuxSwbeG9M3uwKnqhwr6Dv5+SE/9uQiyEAsfRERnNkVxF/dj9cZxwr7CV04Nyv3OiN0FVNk6d9LKTV6cjZyJiJXichWEdkhIp8Y4Ly3i4gRkWW5ii3buis2u5Kz5s4RXzOxq4n6BfcS+VXdiK+lcssYQ/T57QAkDh6n8/HXT16HNUy+qnJCK2tOe574nV6JGYBvziQKrjknq/vA+mZPAsDd3YAxBpP0Mjp6OFKne50SkW+IyIb01zYRae7x2LtFZHv6693ZjrVrBH6gKWyl1NiUk99qEbGBbwNXAwuBm0VkYR/nTQA+Aryci7hyxVdbTnJ/K1gWSGoj6JEwsQRNNz1EYusx2v7t6bx6c1ODYAx2ZTHBC+YSuqyW5LE2EgeOj+iSiYPHh5TguTuPEl2za0T3HA67JIx/8YxU1aibpO3HzxPfdDDncfRlMK9TxpiPGmOWGGOWAPcCv0w/twz4V+B8YAXwryKS1dJsoyNnSo1bufrItQLYYYzZZYyJAw8Ab+njvM8BXwb62CF57HLSFZvJbU34qitGvNVKy/99EveVwwTftgB3wxHiL+7PRJgqR8SyCC6twl8zGV9VOYU3LMM3JzWiNNh9MHsyMZfOP24iun7voJ+TbOogXnfopD5jkRe20/nE60O+/1AFz6vCmVSUKgaAjLbSkJT3i8hTIvJa+tglInLTIJ4+2NepLjcD96f/fCXwpDGmyRhzHHgSuGr4f5NB6Bo5s3XkTKnxJle/1dOAnhnEgfSxbiKyFJhhjPndQBcSkdtEZJ2IrGtoaMh8pFnQs2IzdOkC/AuGv4VN5Jeb6fjmyxR85HxKf/w2pDhAx7fXZipUlUXGGCIvbCdx8ORRsq69LBNHW2n/5bo+e5ANJL7jKCQN/nkDFwL0ZE+aAJ4h2dTefcxri+asSXKyqYPEodSM4KmNaUfos8CtwPeAmeljB4CPD+K5p32d6iIis4Bq4KmhPjdTpMBP8MI5un5VqXEoLz5yiYgFfB2483TnGmO+Z4xZZoxZVlExNpovOnNKwWelKjYB45khTUUmDh5Pbbuz+zjH3/trfMumUvyVVViFAcK3LCHy0CaSR4b2hq5yL7G7AXfrkX6ntVPrwISOx17rs4N/X4wxuNuOYJVPwC4b/Jt0V4XkSYmgm4QcbekUfWkH0Rd3pL7J7Bq3W4BrjTEPAF2/ZLuBTO8PtRr4hTFmSKXXmfxwaQX9+BdMxZqQ2Y3qlVKjL1fJ2UFgRo/vp6ePdZkAnAX8WUT2ABcAj4yXogBxbJyaiSTqGnH3N9H2Py8Muigg2dJJ5xOv0/mHzTT99UMAlP38HYg/VWhb8KHl4Hp0/L+/ZC1+NXJe1CX60k6s8gn4a6f2eY5VFKLgr85B/A6RF3cMKoFP1rfgNXfiP037jF73CvuRwsBJzWiNm8zZfpu+2RUn+nNlNjmzga7hwK4fYGGPYwM53etUT6s5MaU56Odm8sOlibkkj7XrFlhKjUO5Ss7WAjUiUi0iflIvbI90PWiMaTHGlBtjqowxVcBLwPXGmHU5ii/rnIUVuHUNWAV+SHqDTs7iGw8AkHj1GO7aQ5T+4C04s8u6H/fNKydw5Rw6/nNddxNRlX9ia3Zh4klCF9UMWA1phf0EllXjNbbj7qg/7XUTB5uRsH9YW/g4lSUnfW/iie6kP9ucqhM7GNilA7f+GKLfA18XkQCk1qCRWsv6m0E8d8DXqS4isgAoBV7scfhx4M0iUpouBHhz+ljWJA4cp+OR9Xgd42qJrlKKISRnInK5iFSn/zxFRP5bRH4oIqf9yG6MSQC3k3qxqgMeNMZsEpHPisj1ww1+LPHVlpPceRzx+wZdsem1R3F3HMUKhOj49HMU3LGC0A29ilwp+IcVeIfaiP56SzZCVyOUqG/B3XkU/+Lp2GWnT0R8syuwJxXhNUdOe27wvCoK3rJ0WCNeoTfMI3x5bff3ztRS7PIJQ77OcFhBP860UqQwgFWe0TVTHwWmAC1AMakRs1kMYs3ZEF6nVgMPmB5Dm8aYJlJJ4Nr012fTx7Kmq3gkEz3ylFL5ZSgfk+8jVZEE8LX0fyOkFt6eNsEyxjwKPHrKsU/3c+5lQ4hrTHBqK1ILsHc3YxUG8Y6dfpYl1WLA0PbPz+C/bCbhD53b53nBa2qwq0po/9YaQu9YlOHI1XB4nXESB4/jmzMJe1IRoUvmnzRaNBARIXzV2QNW4SVbImA87JKCPnuWDUfoDfMycp3BcmZXkFy3O9WUOd0DcCTSo2TlwI1AGamkbL8x5shgrzGY1yljzGf6ee4PgB8MLerh695uS/ucKTXuDCU5m2aM2SciDqkkbRYQB3TvkEHwpSs23c0NONUTiW86SKK+BWdycf/PqZ1K+91rME0xCr6/kuiLO3BmlPVqxSG2RcGHltP6f5/E3ViP7+zJWf275Lvk8Q4if9yMPakIp7o8NUJjZfcNzLjJVL+ywy0k9jd1J99WSQinoqi7VcZgdSVmyca2VNPYotCJexlD9LlteG1RCm9cPuxWCsYYOh/bmOq5du6sYV1jJHxV5XiN7RnbLsoYY0RkIzDBGHMUOJqRC+erpPY5U2q8GsqrequITAYuBTYbY7qGfjLzsX2cc+ZNBEm10wicO4vAeVWnnUJq/7dniD2wldL/up7AstSbZ9e+hKcKv/dcCDp03KdtNRL7m/Daorj7jxH50xZIpmafvLbIsPqI9WSSHsnGduLb64mu2UWivgWA5LE2On+/kfir+xDbIrB0FgVvOXdE04TGTdLx+EaiL+886bi75TDJo60EllWNqMeViGASHskjLXgdMVp/+gLuztzlM+LYBC+YgxUeWd+/U6wHcjsEOEqMm0xtGp/FHR2UUqNjKCNn95JaS+EH/jF97CJAFzoNgoR82NWlJOoaEZ9NYHGqsMskvV5vsCaWoP2hdXT+7yYK/mF591SlXTEBd+dRAmdP73V9e2KY8M1n0fnjVyn64puwSkK9zjlTJBvbkAlBCt92Ht7xzu71WJ1P1eG1RrCKQt1f9qQifDNSBRZeeyzVGNWkW50Yg/h9WGE/XiRO5+MbU+vAupYa2VZqRGtyMfbEQkKrFmGXT8jYNGPq/5OZxNbtxt3fhG9GGV57jOi6PdhTS4Y8GtcXp2IC8e1HUpWT8WRGG8KOkj8Dj4nIj0j1Heu5LixnU4654Ksuxy4JZ7pPnFIqDww6OTPGfFlE/hdIGmO6PsofBN6XlcjGId/Ciu5eZ3CiTUbowrk4009UYEZf2oVxXZy5ZRR/9c0nnj9nEtGXdpJs6uhzYXnB7Svo/OEGOv/7VQo/ckF2/zJ5zCotwCoOI7aFnV5sbowhsLSK5KHjeK0RvGPtJPY24swq707O2v/3le6u61188ypTFZYBH9aEEM6MidhlBVhlBVgTQt2Vl+Jz8PX4N8wU/8KpuNuOEFuzC2dqCdGXdgCG0MqajLwp25MmQN2hEy01MjTFOIouItXX7NJTjhtyuB4sF+yywiH1tlNKjR1Dqps3xmzr+rOIXA54xpinMx7VOOXUlhN9YicmkUQcG6sgiDg2kee3U/jWpUjAhxeJE990kOSuVkrvuw7pMQrjVFfAut0kG9v6TM78S6fiu2A6HfetpeCOFVlfZ5Wv+lo/JSL4ZpR1J2IAxvNSjVe7nnfhnNRbuKS71osgE1JTbmIJ4Tf1rpTNNrEtAitmE/nDJuKbDmKVhAlMLclY49GuZrRduxaILzetNLLFGHP5aMeQK8mmDvC8nFXYKqVyZyitNJ4WkYvSf/44qX3nfiYin8pWcNUTznQAACAASURBVOONU1sB8STJ3eltaxyL0BvmYSJxoi+nNqFuu/tZJOQQOG8WztyJJz3fCvqYsPqCAbfpKbx9BYltx4j9IfebWucD4yYx3uB2XxDLQgInkl//3Mn4aybjnzsZ35xJ+GZX4KSTl9Hkm1GWqvS0hOCyagILM7crkBQG8M2Z1N3fLFOL80dTutfYu0Tkk+n/ZnUD8tESW7+XyHPbTn+iUmrMGcrQylmkmsMCvB+4nFQn/w9kOqjxqmfFZhe7fAL+c2bi7jxKxy9exfg9TFuCgpuX9HmNrvVT/SUgoXcsxJpUQMe31mQ4+rEhvukgbT97EZMYXw15Q5ctIHBW77WGIyUihC6Zj29+Jb65k08aqR2LRORCYCep16XFwN8DO9PHx5WuEXil1PgzlOTMAoyIzAHEGLPZGLOfVKdsNQjOglSfq0TdyRWXgcUzkMIgsed3k3ztOOG3nD3gdTr/sIno89v7fEwCDgXvX0r0t9tI7Dne5znjWbKxLbU10Th708r2om+rKEjwwrlYIX9W75MDdwMfMsasNMbcbIy5CPgg/P/27jw+yupq4PjvzJJlJiEJYZF9kV1lEQQ31NdaBWuxWqtQF/StS62o6NvWrW5ttWoXcUGrULVarUrVuhSL1K1uIIsIArKD7GuAZDJJZjnvHzOJISSQkNkyc76fTz6ZeebOc8+FzJOT+9yFh5McV+yFwmBrnBmTlpryyf4YeBT4A/AaQDRR2xGHuNKSoyAHR8d8Akvr/JOFlYo/L6biL0so+NV3cHc78GKlkuMmsHZ7g9s1ea4aBg7B93ja7H7VKKoaGY9nY3CaJLh1D2V/n01oy+5khxILfYCX6xz7B9ArCbHElQas58yYdNWU5OxSYDewELgreqwf8FBsQ0pv7gFtCS7Zt+ds76/eo+q9tRQ++X1cvYsbeGetc/RqB8EwwW921vu6q0sBOT/oh2/qfNQfiEncLYGWV6H+QM0MTdM41ZNLymcuTnIkMbGCyPZKtf2IyK3O9BK0njNj0lVTltLYCdxa59i/Yh5RmnP1b0P50wtQVUSEiunLKXvgEzxXDcUz9sC3M6s52xcg3myqVm1rcK2rvGuGU/HKUspf+grvpfVv+3QggQ27cHUsRByOmlhTXWhHKQAO6zlrkpY+Q7OOicBbInIdsA7oDvQGzkpmUPGQc3yvQ9pT1RiT+poyW9MtIneLyGoRqYh+v1tEWvwglURyD2iLllUR2rCX0IY9lFzyGq6B7Sl8cFSjzyEiuA9vF1mzq7yq3jJZp3THNaAtvkc+p9b+zI0S3LoX/8zFVC3dTGDVNvzvL20RA+wdBR6yB3dt1ObiZl9ZAzri6tTyh4+q6qfA4USGYMwjsnh2r+jxtOLqUGi38I1JU03pE38AOI3ILKhB0e+nAvfHIa605eofmbEZXLiVXWP/gVaGaP3yj5Dcps2Sc/dqT/bwng1ueiwieCcMJzB/M4HZGxp9XlWlcu4aJNdNVp/D0MogwXU7KZ/xFeGK1L5F6iz0kD2km43DOQQ5Iw7Hc/qRyQ6j2USkE4Cq/k1VH1DVvxGZyNQxyaHFXGDdDkJ7ypMdhjEmDpqSnP0IGKOq76jqMlV9BzgHOD8+oaUnV//IYP/d106n6pP1FD5xFu6+B54AUB9nQS7ZAzohWS6CG0sIbtp/ZqbnooFIfhZlkxu/32bwm52RfRuHdEPcTrIGdCT3f/pH9o7815eESyuaHGsiqCrBTbvRqmCyQzHJ9U+g7pojnYlOYkoXGlb87y0l2MBeu8aYlq0pyVlDg45SfzBSCnG09eIoziW0ZjeeK47G8+OBzTqfqlK5cD3lM76iYt7afdY/c+Rn47l0MP6XFxPaWnaAs0TPFVYq563FUZCLu/e3C926u7fBc8ZRhCsC+N5akJI9aFpWQfmMRQ1uDG8yRh9VXVT7QPR5vyTFEx+h6DAD6yU2Ji01JTmbBrwpImeISH8RGUXkr9Rp8QktPYkI7uGdcA9qT+FDo2NyPs9pR+DucxhVC9dT/vZCwmXf9m55f3YMVIUonzr/oOfS8kpwCNlDe9TsGVnN1b4A7/cGkXVU55qNvf0ffk3lgnUEN5YkvccqtCOSfNoYnIy3XUT2WTYj+rz+qc0tlAbDAIjbZmsak46aMk3rl8CvgMlARyKbnr8I/CYOcaW14lcuAGjyOLOGiNtJ7gm9cXUowP/pSspen0/+ecORbBfufm3J/m5PfH+eS95NJxxwPJYjLwfvmKMb7At1FnpwFnoA0KogoV2+fXqqnO1akT20O67DCmLSrqYI7SgFh+Ao8iS8bpNSngJeEZHbgNVEJgf8Fpia1KhirXqCjtN6zoxJRwdMzkTk1DqHPoh+CZEtogFOBN6LdWDpLFZJWV3unu1wts2navEmJDvyX+v/dAXZ4/sR+uPn+F//Gs8Pj6j3vcFNJTiL82vedzCS5SLvnKGRJG1HKaFte6lasbWmB02DIXA49uuBi5fQjjIcrfMQp/UkZLj7gACRxbK7AN8QScweTGZQsWY9Z8akt4P9Jv5LA8erE7PqJK1nzCIyzeLIzyXn2MO/PSCCEiL32kEEdu3A//Fysvp12Of2X9hfRfm7S3B3b0PuyL5Nqk+yXLg6FuHqWETWwC4QXQ+t8sv1BNftIOuoLrgPb4s44vdLRFUJ7SxrcM03k1FOBv6hqr8XkQ5EZpMfCbQDtiQ1shhy5OXgOXMgjoLcZIdijImDAyZnqtojUYGY+Mg9rhc5I3pSOukTAgs3I1lOJMeNs01+ZPJAMETlgm8gpJHkqhlqJ2DOtvkEN+yi4uPlBFZuxXPGkYecoKkqoW2lONs03DPmHXWUDY42AI8BZ0Qf/zH6PQA8CYxJSkRxIG4nrvaJHz5gjEmMtFoa3NRPHA68lxxNaecPcOXnkz12BADBjSX4P1gKoTDuvh1wFsRuvJa7azGuLq0JLNtCxWcrqZy7lpzhTe9g1XCYis9WEVi+BXefw8g9ofd+ZUTEJgKYap1U9RsRcQGjgK5AFbApuWHFVrisguCWPbi7tEay4zNMwhiTPDZgIUM423jxjD0ysnVURWQwsSM/B3ePtjiK88ge3DXmdYoIWf064O7fgcDKrYQr6t/N4ECqlm4msHwLjtZeAsu3ENy8/+bcgXU7CKzdUc+7TQbaKyLtidzeXKyq1WvIpFUGE9pWSsVHywn7m/6ZMsakPus5yyDeCcMp/+uXlD/7JXnXjsBZ6CH3xD5xrzfnmJ7oUV1w5DR+p6/q/Tyz+nfA0SoHV4dCyv45n8ovv8HVoXCfslWLNoBDcHdv+mK+Ju08AswBsojsswlwAvB10iKKg+rt1Gw3DGPSk/WcZZCsYZ1wD++Eb3LT99tsDnE6cHizUVWqlm5CAwdeEy20ozSyXltFFeJw4O5SjLiceE4bgOc7A/Ypq2EltMtntzUNAKp6P5Ft5k5Q1RejhzcClycvqjiIztakge3bjDEtm32yM0zehOEEl+2k8t3VCa87vMtHxexV+D9eUW9yqGElsHY7vrcXEi6rRCv2TeKchV7E7UJD4ZqFdsO7fRAK4yzOS0gbTOpT1eWquqrO80UHek81ERklIstEZKWI3NxAmfNFZImILBaRF2odD4nIgujXG81vScM0ZD1nxqQzu62ZYXJ/NIA9/zcD36Ofk3Pa4Qd/Qww5i/PIHtqdyrlrqVqyiewjOtW8VjF7FYHV29GKAI42eXi+cwQOT/23QctnLkYrA3i/P7hmZwCH9ZyZZhIRJ5FFtr8LbADmiMgbqrqkVpnewC1EeuZKRKT2+i1+VR2ciFir1znD1vUzJi3ZJzvDSI4bz+VHU/HmcoLr9h9cH29ZR3bG1bWYyjmr8f93Wc1xDSvOjoXkntwX7+hBDSZmAFn9OhDe5aPqq42Ed5dDlhNHq5xEhG/S23BgpaquVtUqIjugnF2nzBXAZFUtAVDVbQmOEYh8BrxjhiBiWxsbk44sOctA3p8OA8D357kJr1tEyB3ZB0eRl9COUrQycusy97heeE7uh7tnO+Qg42jc3dvg6lZM5YJ1uPseRt4Pj7FfUiYWOgHraz3fED1WWx+gj4h8IiKzonsMV8sRkbnR4z+orwIRuTJaZu727dvrK9Iojtwsu5VvTBqz5CwDuboWknN2X8qnzEMrAgmvX7Jc5J19NHnnDmv0dlF15RzbC5xOKj5decjnMOYQuIDewCnAOGCKiFRPH+6mqsOAHwOTRGS/cQOq+qSqDlPVYW3btj3kIIIbdhFYlZROO2NMAiQsOTvYQFsR+amILIoOpv1YRAbUdx4TG95rhhPe6cf/8uJkh3JIHJ4scoZ2I1ziQ0srkh2OSQ8biezHWa1z9FhtG4A3VDWgqmuA5USSNVR1Y/T7aiJ7EA+JV6BVy7dQuXD9wQsaY1qkhCRntQbajgYGAOPqSb5eUNWjogNqHwD+lIjYMlX2qT1w9WtD2aOfJzuUQ+bu2wHPd49E8m28mYmJOUBvEekhIlnAWKDurMt/Euk1Q0TaELnNuVpEikQku9bxE4AlxEswbMtoGJPGEvXpPuhAW1XdW+upl283VzdxICJ4JwwnMGcTVZ9vSHY4h0REcLbNt/FmJiZUNQhMAGYAS4GXVXWxiPxaRKr35ZwB7BSRJcD7wC9UdSfQH5grIl9Gj99Xe5ZnzGMNhhrcZ9YY0/IlarBOfQNtR9QtJCLXADcSWd371PpOJCJXAlcCdO0a+y2HMonn4oHsvfk/lE2eQ+vhnZMdjjFJp6rTgel1jt1R67ESuUbdWKfMp8BRiYgRQENhHLanpjFpK6X+9FLVyap6OHAT8KsGysRkQK0BR6scPOMH4f/7InZPfJuqBZuTHZIxpjECIbutaUwaS1TPWWMG2tb2IvB4XCMyAOTfcTKhrT58j8/F99Bs3IPa47l0MLkXDsTZ1pvs8Iwx9fCMGpjsEIwxcZSoP70OOtA2uvJ2te8BKxIUW0ZztsujeNr5HLbp/yh49ExwO9lzwwy2dPwjO895Ef/rX6OBULLDNMbU4vBkHXChZmNMy5aQnjNVDYpI9UBbJ/BU9UBbYK6qvgFMEJHTgABQAoxPRGwmwlnsIe+a4eRdM5zAoq2U/3UB5c8tpOKfX+No6yH3ooF4Lx2Me+BhyQ7VmIxXuXA9znb5uA4rPHhhY0yLI/VtQN1SDBs2TOfOTfwq95lCAyEqZqyk/OkFVLy5DAJh3EMOi9z2/PFRONvYbU+TeCIyL7rYa4t2qNcvVaX0mY/JGtyVnCHd4hCZMSZeGnv9shGlpkHidpJ7Vl+KX7kgctvz4dEgwp7r/x257Xnui/jfXGa3PY1JpFBk0/ODbXNmjGm5bN8b0yjONl7yrh1B3rUjCCzcQvlfv6T8bwupeO1rHO28eC4aiHfCcFw9ipIdqjFpTYORP4bE6UxyJMaYeLE/vUyTuQceRsEfz+CwDTfS+vWxZJ3QhbKHZ7Pj5KfRqmCywzMmvQUjPWe47fJtTLqyT7c5ZOJ2kjumH8WvjqX4zXGE1u+l/Nkvkx2WMWnNes6MSX+WnJmYyD6jF+6hHSi97+OaXx7GmNhzFHjI//GxuLoWJzsUY0ycWHJmYkJEyL/tJEKrSvC/tDjZ4RiTtkQEyXbbhABj0ph9uk3M5JzdF9cRbSm99yM0HE52OMakpdBuHxVz1xD2VSY7FGNMnFhyZmJGHA7ybx1JcMl2Kl5fluxwjElL4d3lVC3agFba5Btj0pUlZyamcs8/AufhRZTe819a8gLHxqQqDdg6Z8akO/t0m5gSl5P8m08kMG8zle+sSnY4xqSfUHTCjctmaxqTriw5MzHnuWQQzs6tKL3nv8kOxZi0U9NzZuucGZO27NNtYk6yXOT94niqPvqGyo/WJTscY9JLdPsmbJ0zY9KWJWcmLjyXH42jrSdpvWehrWVJqdeYeMse3JX8S05AHJLsUIwxcWLJmYkLhyeLvBuPo3LGKqrmbExo3aV/+pQtHf5A5QdrElqvMYkiTrt0G5PO7BNu4sb7s2OQwhxK7/0oYXUGvtrK3lveBYXS+z5OWL3GJErV8i1UzF+b7DCMMXFkyZmJG0erHPKuG0HFP78m8NXWuNenVUFKLn4NR0E23huOpXLGKgJfbol7vcYkUnBTCcE1O5IdhjEmjiw5M3HlvW4E4nVT+rv492KV/vpDAgu2UDhlDK3uOBnJy6L095/EvV5jEioYBlvjzJi0Zp9wE1fOYg/eq4/B/+JXBFfujFs9VbPWU/q7j/FcNpjcs/vhKMzFe9XQSL3rdsetXmMSTYMhxG0zNY1JZ5acmbjLu/E4cDsovT8+vVhhXxW7LnkNZ5dWFEwa9W29E48FEcr+9Flc6jUmKYJhsAkBxqQ1+4SbuHN2yMf7k6Mp/+sCguv3xPz8e2+aSWjFLoqe+QGOVjnf1tu5AM+FR1E+dT6hneUxr9ekHxEZJSLLRGSliNzcQJnzRWSJiCwWkRdqHR8vIiuiX+PjGmeWK56nN8YkmSVnJiHyfnkCKJT94dOYnrfinZX4Js/Be8OxZJ/SY/96f348Wh7A99icmNZr0o+IOIHJwGhgADBORAbUKdMbuAU4QVWPACZGj7cG7gRGAMOBO0WkKB5xes8ajOd/+sfj1MaYFGHJmUkIV7dCPBcPxPfkvJgtEBsu8VPyv6/j6t+Ggnu+U28Z95Htyf5eb3wPz0b9gZjUa9LWcGClqq5W1SrgReDsOmWuACaragmAqm6LHj8DmKmqu6KvzQRGYYwxh8CSM5MweTefCFUhyh6MzRiw3ROmE97qo+i5c5Fcd4Pl8n95AuEd5fieWRCTek3a6gSsr/V8Q/RYbX2APiLyiYjMEpFRTXgvInKliMwVkbnbt28/pCD9/11GYPW2gxc0xrRYlpyZhHH3aUPujwbge2wO4RJ/s85V/vJX+F9YRP7tJ5E1tOMBy2aN7Ib72M6U/eFTtHpfQmMOjQvoDZwCjAOmiEhhY9+sqk+q6jBVHda2bdsmV66qBFZvI1RiYyiNSWeWnJmEyr91JFpaRdkjsw/5HKHNpey++l+4j+lI/i0jD1peRMj/5QmEVpfgf2XJIddr0t5GoEut552jx2rbALyhqgFVXQMsJ5KsNea9zRdWUBBb58yYtGafcJNQ7oGHkfP9PpQ9NJtwWWWT36+qlFz+BloeoOjZcxq93lPOmL64+hRT9sAnqGqT6zUZYQ7QW0R6iEgWMBZ4o06ZfxLpNUNE2hC5zbkamAGcLiJF0YkAp0ePxVYwFPnusnXOjElnlpyZhMu/7SR0lx/fn+c2+b3lU+dTOX0FBfefhrtf428LidNB3s+PJzBvM1Xv24boZn+qGgQmEEmqlgIvq+piEfm1iIyJFpsB7BSRJcD7wC9Udaeq7gJ+QyTBmwP8OnostjEGI7flrefMmPRmn3CTcFkjOpN9Wk/K/vhZk2ZQBlfvYs8N/yb7Oz3wThje5Ho9Fw/E0d5L6QO2pZOpn6pOV9U+qnq4qt4TPXaHqr4RfayqeqOqDlDVo1T1xVrvfUpVe0W/no5LgGFFPFlItq1zZkw6s+TMJEX+bSMJbynD99QXjSqvoTAl4/8JLgeFT/8AcTT9R1dy3ORdbxuim5bLkZ9D/gUjcHdv+mQCY0zLkbDk7GArb4vIjdFVtxeKyLsi0i1RsZnEyzq5O1nHd4mMAQuEDlq+7E+fUfXxNxQ+ciauLgWHXK/36mG2IboxxpiUlpDkrDErbwNfAMNUdSDwD+CBRMRmkkNEyL9tJKFv9lD+t4UHLBtYtJW9v3qPnHP7k3vRwGbVu8+G6GtLmnUuYxIttKOU8plfEdpjS2kYk84S1XN20JW3VfV9Va2+4swiMhXdpLHs0b1xDzmM0t991OD6Y1oVpOTiV3EU5lD457MQkWbXW7Mh+oOzmn0uYxIp7KskuKEEbL0+Y9JaopKzRq2eXctPgLfjGpFJOhEh/9aRhFbswv+P+tcf23v3hwS+3ErhlO/jbOuNSb22Ibppqapna+K0pTSMSWcpNyFARC4ChgG/b+D1Zm9/YlJHzrn9cfVvQ+m9H+23/ljlZ+spu+9jPP87hNwx/WJar22Iblqk6DpntpSGMektUZ/wRq2eLSKnAbcBY1S13hVKm7v9iUkt4nCQf8tIggu3UvHW8prjYV8VJZe8hrNLKwoePCPm9dqG6KYl+nadM+s5MyadJSo5O+jK2yIyBHiCSGJmu/pmkNxxR+LsUUjpPf+t6T3b+4t3CK3aRdFfz8HRKicu9dZsiP5045bzMCbZxOXA0SoXrOfMmLSWkE94I1fe/j2QB0wTkQUiUnfbFJOmxOUk/6YTCczeSOW7q6mYsRLf43PJu/E4sk/uHrd6azZE/+NnaPDgy3kYk2xZfTuQ98NhiNOSM2PSWcKWmVbV6cD0OsfuqPX4tETFYlKP59LB7P31h+y9/X1C3+zBNaAtrX57alzrrN4Qfde5L+F/dSme84+Ma33GGGNMY9ifXyYlSLaL/J8fT2DWBsLbfBQ9dw6S4457vbYhumlJKr/8hvL36p/ZbIxJH5acmZThuXIo7qM7UPCH08k6umNC6rQN0U1LEtpdTmiXL9lhGGPizJIzkzIc3izazbuKvOuPTWi9tiG6aTGCYZupaUwGsOTMZLzaG6JXLdic7HCMaZAGQ7bGmTEZwD7lxvDthuhlv/802aEY07BgGKznzJi0Z8mZMdTaEP0l2xDdpC5Hq1ycRZ5kh2GMiTNLzoyJsg3RTarLHdmHnBGHJzsMY0ycWXJmTJRtiG6MMSYVWHJmTC22IbpJZb63FlC5aEOywzDGxJklZ8bUUntD9HB5VbLDMWYfoZ1laGUg2WEYY+LMkjNj6qjeEL38mQXJDiXmyh6ZzbZhT6AB20u0pdFwGMJq65wZkwEsOTOmjlhsiK6qhPdUEFy1i6p5m1KiF04DIUrv+5jAvM34py1OdjimqYLhyHdb58yYtJewjc+NaSnqboiee25/wiUVhHeUE95ZTninP/J9R63HdY/t8n/7yxTwjB9E0TPnJLFVUPHGMsKbShGPm7IHZ5E77ihEJKkxmcbT6M+T9ZwZk/4sOTOmHtUbopdc+ColtZKs/bgdONp4cBR7cBTn4urfFkdxbuR5m8ixyukrKH9hEa3uOw3nYfmJa0Qdvsfm4OxaQN4vjmfPtW9T9el6sk/omrR4TNM5OxbiyM9JdhjGmDiz5MyYeojTQeHUMfinLcbRet9kqzoRcxR7kLysg/Y+ZZ/QFf+0JfiemEerO09JTAPqCCzdTuV7a2h173fwXDaEvbe/T9mkWZac1SEio4CHACcwVVXvq/P6pcDvgY3RQ4+q6tToayFgUfT4N6o6JpaxOTxZeM84KpanNMakKEvOjGlA9shuZI/s1uzzuHoXk31mb3yPzyH/5hOR7MR/7Hx/ngtuB56fDMHhzcJ75VDK/vApwXW7cXUrTHg8qUhEnMBk4LvABmCOiLyhqkvqFH1JVSfUcwq/qg6Od5zGmPRnI0uNSYC860cQ3urD/3LiB+KHfVWUP7OA3B8dgbNdHgDeCcNBwPfI7ITHk8KGAytVdbWqVgEvAmcnOaYawS17KJ32OaEdpckOxRgTZ5acGZMA2d89HFf/NpQ9NAtVTWjd/hcWoXsr8f7smJpjri4F5J43AN/U+YRLKxMaTwrrBKyv9XxD9FhdPxSRhSLyDxHpUut4jojMFZFZIvKD+ioQkSujZeZu3769ScFpVRAts/8rYzKBJWfGJICI4L12BIF5m6n6bP3B3xAjqorvsTm4BrYn6/gu+7yWd8Nx6J7KtFzPLY7eBLqr6kBgJvDXWq91U9VhwI+BSSKy3yaYqvqkqg5T1WFt27ZtWs3Vy7rYbE1j0p4lZ8YkiOeSQUhhDr6HEncrseqz9QQWbCHvZ8fsN3Eha0TnyHpuD82KLHBqNgK1M9jOfDvwHwBV3amq1d1XU4GhtV7bGP2+GvgAGBLL4L5dSsMu28akO/uUG5MgDm8W3suPxv/KEkIb9iSkTt9jc5BW2eReWP8sv7wbjiW0qoSKt5YnJJ4UNwfoLSI9RCQLGAu8UbuAiHSo9XQMsDR6vEhEsqOP2wAnAHUnEjSLWs+ZMRnDkjNjEsh7zTGgUJaAjdVD28rwT1uCZ/wgHHnZ9ZbJPbc/zi6tKJs0K+7xpDpVDQITgBlEkq6XVXWxiPxaRKqXxbhORBaLyJfAdcCl0eP9gbnR4+8D99Uzy7NZHHk5uLoW2yK0xmQAW0rDmARydS8i5+y+lD85j1a3n4zkuuNWV/lTX0BVCO/VxzRYRlxOvNeOYO8vZ1K1YDNZgzs0WDYTqOp0YHqdY3fUenwLcEs97/sUiOsiZO6uxbi7FsezCmNMirCeM2MSLO/6Ywnv9FP+wqKDFz5EGgrj+/Ncsv6nO+7+Bx547r38aMTjTuhYOGOMMQ2z5MyYBMs6qRuuge3juqxGxdsrCK3bQ97PGu41q+YoysVz2WDKX1hEaEti19Dyv7kMDdlkhMaomL2KslfnJjsMY0wCWHJmTIKJCHnXjyC4aBtVH6yNSx2+x+bg6JBHztn9GlXee90IqApFdhJIEP8rS9g15u+UT52fsDpbMq0MoqHErpFnjEkOS86MSQLPj4/C0cZD2cOxv5UYXLWLyn+vxHvlUMTduMHj7j5tyDmrD77H56IVgZjHVFdocym7r3oT99AOeP43pitOpC0NhmwZDWMyhH3SjUkCyXHjuXIoFa9/TXBNSUzP7XtiLjgE7xVDD164Fu/EYwlv81H+969iGk9dqkrJ5W8Q9gUoeu7cRieQmU6DYVtGw5gMYcmZMUmS97NjwCH4Hv08ZudUfwDfX74g5wf9cHZq1aT3Zp/aA9dR7SibFN8tpsqnzKNy+goK7j/toJMVTC3Wc2ZMxrBPujFJ4uzUKrK/5V/mE47RnonlLy9Gd/nJu2Z4k98rIuRNPJbgwq1Uvb8mJvHUFVy5kz03zCD7Oz0im6+b4o2vfwAAGE5JREFURnN1KsLVuXWywzDGJEDCkjMRGSUiy0RkpYjcXM/rJ4nIfBEJish5iYrLmGTyXn9sZH/LZ7+Myfl8j83B1a8NWad0P6T3e358FI62nrgsSqvBECWXvAZZToqe+QHisL8NmyJ7UFeyj+qc7DCMMQmQkKujiDiBycBoYAAwTkQG1Cn2DZHVtl9IREzGpIKsYzvjPqYjvodnN3t/y6q5Gwl8vhFvPftoNpbkuPFefQwVby0nuGJns+Kpq+yBT6j6bAOFk8/E2bkgpufOBPG81WyMSS2J+tN1OLBSVVerahXwInB27QKqulZVFwK26JHJGCJC3nUjCC7bSeU7q5p1Lt/jcxGPG88lg5p1Hu/Vw8DtjOlM0qr5m9h75wfknn8EuePiupB+2ir7+ywqZjXvZ8QY0zIkavumTsD6Ws83ACMO5UQiciVwJUDXrl2bH5kxSZZ7/hHs+cVMyh6eTc6o3od0jnBJZMcBzyWDcBTkNCse52H5eMYdSfnTX9DqN/+DozC3WedTf4CSi1/D0dZD4ePfO+RevUynwTA47VZwrO3du5dt27YRCMR/CRmT3txuN+3ataNVq6ZNxqpPi9tbU1WfBJ4EGDZsmPXzmxZPslx4rx5G6Z0fEFi2A3ffNk0+R/kzC6AiGOn1igHvxGMp/+uX+KbOJ//nJzTrXHtufZfgku0U//siHK09MYkv02hYIRQGm60ZU3v37mXr1q106tSJ3Nxc+8PBHDJVxe/3s3HjRoBmJ2iJ+qRvBLrUet45eswYA3ivGgZZTnyPNP1WoobDlD0+h6zju8Rs4/KswR3IOqU7vkc+R4OhQz5P5Xur8U2ahfdnx5BzRq+YxJaRoltcia1zFlPbtm2jU6dOeDweS8xMs4gIHo+HTp06sW3btmafL1HJ2Rygt4j0EJEsYCzwRoLqNiblOdvnkTv2SMqfWUB4T0WT3lv57hpCK3bhbcQ+mk2RN/FYQt/soeK1rw/p/eHdfkou/SeuPsW0euC7MY0t01QnyLbOWWwFAgFyc5t3296Y2nJzc2Nyizwhn3RVDQITgBnAUuBlVV0sIr8WkTEAInKMiGwAfgQ8ISKLExGbMaki77oRqC9A+VNfNOl9vsmf42jrIfe8uhOgmyfnrD44Dy865GU1dl/7NqFNpRQ9dw4Ob1ZMY8s04hCyjuiEo3VeskNJO9ZjZmIpVj9PCfszTFWnq2ofVT1cVe+JHrtDVd+IPp6jqp1V1auqxap6RKJiMyYVZA3tSNYJXSh7ZDYaatyk5eA3u6l4czmey49GsmM7hFScDvKuG0HVp+up+nxDk97rn7YY/98Wkv+rk8gabmtzNZdku8kZ3hNX++YPNDbGpD7rIzcmheRdfyyhNbup+NfyRpUvf3IeqEbGrMWB57IhSKvsJvWehTaXUvLTt3AP60j+bSfFJa5Mo+EwGgjZWmfGZAhLzoxJITnn9MPZuRW+hw4+MUCrgvimzCfnrD64uhXGJR5Hfjbey4/GP20JoQ17Dh6TKiU/eR0tD1D03Dm2qXmMhLbupfRvnxLacvD/A2NMy2fJmTEpRFxOvNccQ+V7awgs2nrAsv5XlxLe5ov5RIC6vNcOh7BSNnnOQcuWPzGXyrdXUvD77+LuZ5uax4oGbbam2dell16KiCAiuN1u2rRpw4knnsgDDzyAz+dLdnimmSw5MybFeK4YiuS6KDvIshq+x+bg7FlE9umHxzUeV/cics7ph++JuYR9VQ2WC67YyZ7/e4fs7/aMe8KYcaqXM7HZmqaWkSNHsnnzZtatW8f777/PhRdeyKOPPsrRRx/N1q0H/uMunVVVNXydainsk25MinEWe8i9aCDlzy0ktLO83jKBRVup+ugbvFcPS8gG4nk3HIeWVOB/rv4N2jUYYtfFr0Y2NX/aNjWPNes5M/XJysrisMMOo2PHjhx11FFcffXVfPbZZ2zfvp2bb755n7KPPPII/fr1Iycnh969e3PPPfcQDAYBuO222+jbt+9+57/66qs58cQTG6x/5syZnHLKKbRu3ZqCggJOPvlkPv/8833KlJWVMXHiRLp06UJ2djbdu3fn3nvvrXl927ZtXHbZZbRv356cnBz69u3LU089BcAHH3yAiLBhw74TklwuF8888wwAa9euRUR4/vnnOfPMM/F6vdx+++2oKldccQWHH344ubm59OzZk1tvvZXKysp9zvWf//yHkSNH4vF4atqwatUqPvjgA5xOJ+vXr9+n/LPPPktBQUHceydb3A4BxmSCvGtHUD5lPuVT5pF/88j9Xvc9PgdyXHguG5KQeLKO74J7WEfKJs3Cc+XQ/ZKv0vs+JjB7I0V//yHOTjajMOas5yyhfG8v3O+Yu3sbsvp3RIMhymfuv9KTu1d7snq3J1wRwP/+0v1ez+rbAXfPtoTLKvF/tGy/172jB8Yk9k6dOnHhhRfy7LPP8pe//AWHw8Fdd93F008/zaRJkxg8eDBLly7lpz/9KRUVFfzmN79h/Pjx3HvvvcyePZsRIyI7K1ZWVvLSSy9x3333NVhXWVkZP/vZzxg0aBDBYJAHH3yQUaNGsWLFCoqLi1FVzjrrLL755hseeeQRBg4cyIYNG1i2LNJ+v9/PySefTG5uLs8//zw9e/Zk5cqV7Nq1q8ntvummm7j//vuZPHkyEBn/2q5dO1544QXat2/PwoULueqqq3C73dx9991AJDE744wzuPbaa3n00UfJzs7mk08+IRAIcMopp9C7d2+eeuop7rzzzpp6pkyZwo9//GO8Xm+TY2wKS86MSUHuo9qTfWoPfJPnkPfz4/fpMQnvraD8uYV4xh6Jszgx2yGJCHkTj6XkolepnLGKnNHf7gFaNW8TpXd/SO7YI/GMtU3N48HRJo+sQV0Qt12yzcEdccQR7N27lx07dpCXl8cDDzzAq6++yqhRowDo0aMHv/3tb7nuuuv4zW9+Q58+fRgxYgTPPvtsTXL25ptv4vf7Of/88xus55xzztnn+ZNPPskrr7zCv//9by688ELee+89PvzwQ+bMmcOwYZEZ5T179uSkkyKzuF944QXWrFnDypUr6dy5c83rh+Kqq67iwgsv3OfYPffcU/O4e/furFq1iscee6wmObv77rsZPXo0kyZNqinXr1+/msdXXnklDz30ELfffjsOh4Ovv/6ajz/+mIcffviQYmwK+6Qbk6K8149g19kvUvHa1+T+6Ntl/8qfW4iWVSV8XFfujwaw55czKZs0qyY5U3+AkotexdHeS+HkMxMaTyZxtW2Fq631SCbKgXqxxOU84OuOHPeBX8/LjlkvWUOql1wRERYvXozf7+eHP/zhPgukhkIhKioq2L59O23btmX8+PHcfvvtTJo0CbfbzbPPPsuYMWMoLGx4JviaNWu44447+Oyzz9i2bRvhcJjy8nLWrVsHwLx58ygqKqpJzOqaN28eAwYMqEnMmmP48OH7HZsyZQpTp05l7dq1+Hw+gsEg4fC3a0jOmzfvgD2D48eP57bbbmPGjBmMHj2aqVOnMnToUIYMif8dC+sjNyZF5XyvD84ehZQ99O0aY6qK77E5uId1JOuYTgmNR7Jc5F1zDJXvrCKwOLJ33J6b/0Pw6x0UPf2DtNjUXERGicgyEVkpIjfX8/qlIrJdRBZEvy6v9dp4EVkR/Rofy7i0MkDY3/IHOZvEWLx4MQUFBRQXF9ckI9OmTWPBggU1X4sWLWLFihW0bt0agLFjx1JaWsq//vUvtm/fzr///W/Gjz/wj3H1LcvJkycza9YsFixYQLt27WI2IN8RHT5Re32/UCi0T4JVre5txmnTpnHNNddwwQUXMH36dL744gvuuOOOJm2tVFxczHnnnceUKVOoqqri2Wef5corrzzE1jSNJWfGpChxOsi7dgRVn6ynat4mAKr+u47gku1Jmw3puWoY5Lgoe2gWFf9Zhe/h2XgnDCfnu/GdMZoIIuIEJgOjgQHAOBGpb0+sl1R1cPRravS9rYE7gRHAcOBOESmKVWwV89fhe21erE5n0tjGjRt5/vnnOffcc3E4HBxxxBHk5OSwevVqevXqtd+X0xkZMlFUVMT3v/99nnvuOf7+97/TunVrzjjjjAbr2blzJ0uWLOHmm2/mjDPOYMCAAeTk5Oyz6ffQoUMpKSlh7ty59Z5j6NChLFmyZL8B/9XatWsHwKZNm2qOLViwoFGLMf/3v/9lyJAh3HjjjQwdOpTevXuzdu3a/ep/5513Dnieq666ijfffJMnnngCv9/PuHHjDlp3LFhyZkwK8/zvEMTrpuzhyLIavsfmIEU55F6QnN3NnMUePJcMovy5hZFNzfsW0+r+05ISSxwMB1aq6mpVrQJeBM5u5HvPAGaq6i5VLQFmAqNiFlkwBDZT09RRVVXFli1b2LRpE4sWLeLxxx/nuOOOo127dvzud78DIC8vj1tvvZVbb72VyZMns2zZMhYvXsyLL77ITTfdtM/5LrnkEt566y3+/Oc/c+GFF9YkbvUpKiqibdu2TJkyheXLl/PZZ58xbty4fTaSP/XUUxk5ciQXXHABr7/+OmvWrOGTTz5h6tSpAIwbN45u3boxZswY/vOf/7BmzRreffddXnrpJQB69epFt27duOuuu2rGe91www2N2r+yb9++LFq0iNdff51Vq1bx0EMP8eqrr+5T5vbbb+ftt99m4sSJLFy4kGXLlvHMM8/UTFgAOPHEE+nbty8///nPGTt2LPn5+QetOxYsOTMmhTkKcvBcOhj/i19RtWAz/leX4r1sCA5P8jYSz7t+BFQECW8po+i5c5MaS4x1AmrPm98QPVbXD0VkoYj8Q0S6NOW9InKliMwVkbnbt29vdGAaDCM2U9PU8dFHH9GhQwe6du3KKaecwvPPP8+ECROYP38+7du3ryl3++2386c//YkpU6YwaNAgTjzxRB588EG6d+++z/lGjx5NQUEBS5cu5ZJLLjlg3Q6Hg2nTprFq1SoGDhzIpZdeysSJE+nQoUNNGRHhX//6F2eeeSY//elP6du3LxdddBE7duwAwOPx8OGHH3LkkUcyduxY+vfvzzXXXIPf7wciS2a89NJLbNu2jSFDhnDNNddwzz331NzuPJCrrrqKiy++mMsuu4whQ4Ywe/Zs7rrrrn3KnH766UyfPr1mlurw4cP561//itvt3qfcFVdcQVVVVcJuaQJIS96rbdiwYdpQd6kx6SKwbAfb+j2Ks3MrQhv20n75tbh6Fyc1pr13vY+zWyHeBC3lUZuIzFPVmG8mKiLnAaNU9fLo84uBEao6oVaZYqBMVStF5CrgAlU9VUR+DuSo6m+j5W4H/Kr6h4bqa8r1q3zmV4T9AfLGJP7fO50tXbqU/v37JzsMk+J++ctfMnPmTL744otGlT/Qz1Vjr182W9OYFOfu24bs0b2ofHsl2acfnvTEDKDVXf+T7BDiYSPQpdbzztFjNVR1Z62nU4EHar33lDrv/SBWgVnPmTGJt2fPHpYvX86TTz6ZkOUzarNPuzEtQN6Nx0W+Xz8iyZGktTlAbxHpISJZwFjgjdoFRKRDradjgOrVRmcAp4tIUXQiwOnRYzGR1a8DWf07xup0xphGOPvssznppJM455xzuOiiixJat/WcGdMC5Jx2OIdtvBFnR1vrKl5UNSgiE4gkVU7gKVVdLCK/Buaq6hvAdSIyBggCu4BLo+/dJSK/IZLgAfxaVZu+zHkD3D1sE3ljEu2DDz5IWt2WnBnTQlhiFn+qOh2YXufYHbUe3wLc0sB7nwKeimuAxpiMYLc1jTHGZKyWPCnOpJ5Y/TxZcmaMMSYjud3ummUbjIkFv9+/31Ich8KSM2OMMRmpXbt2bNy4kfLycutBM82iqpSXl7Nx48aanQ2aw8acGWOMyUitWkXGcW7atKlJey4aUx+320379u1rfq6aw5IzY4wxGatVq1Yx+WVqTCzZbU1jjDHGmBRiyZkxxhhjTAqx5MwYY4wxJoVYcmaMMcYYk0IsOTPGGGOMSSHSktd2EZHtwLomvKUNsCNO4aQaa2t6ypS2Hqid3VS1xW82eYDrV6b8H1fLtPZC5rU509oLDbe5UdevFp2cNZWIzFXVYcmOIxGsrekpU9qaKe2sT6a1PdPaC5nX5kxrLzS/zXZb0xhjjDEmhVhyZowxxhiTQjItOXsy2QEkkLU1PWVKWzOlnfXJtLZnWnsh89qcae2FZrY5o8acGWOMMcakukzrOTPGGGOMSWkZkZyJyCgRWSYiK0Xk5mTH01wi8pSIbBORr2oday0iM0VkRfR7UfS4iMjD0bYvFJGjkxd504lIFxF5X0SWiMhiEbk+ejzt2isiOSLyuYh8GW3r3dHjPURkdrRNL4lIVvR4dvT5yujr3ZMZ/6EQEaeIfCEib0Wfp21bDybdrlPVMul6BZl1zaqWidcuiO/1K+2TMxFxApOB0cAAYJyIDEhuVM32DDCqzrGbgXdVtTfwbvQ5RNrdO/p1JfB4gmKMlSDwf6o6ADgWuCb6/5eO7a0ETlXVQcBgYJSIHAvcDzyoqr2AEuAn0fI/AUqixx+MlmtprgeW1nqezm1tUJpep6o9Q+ZcryCzrlnVMvHaBfG8fqlqWn8BxwEzaj2/Bbgl2XHFoF3dga9qPV8GdIg+7gAsiz5+AhhXX7mW+AW8Dnw33dsLeID5wAgiCxm6osdrfp6BGcBx0ceuaDlJduxNaGNnIr+kTgXeAiRd29qIf4u0vE7Vak9GXq+ibciIa1at+NP+2hWNO67Xr7TvOQM6AetrPd8QPZZu2qvq5ujjLUD76OO0aX+0K3gIMJs0bW+0m3wBsA2YCawCdqtqMFqkdntq2hp9fQ9QnNiIm2US8EsgHH1eTPq29WBa9M/tIUjLz29dmXDNqpZh1y6I8/UrE5KzjKOR9DytpuGKSB7wCjBRVffWfi2d2quqIVUdTOSvsuFAvySHFBcichawTVXnJTsWk1zp9PmtLVOuWdUy5doFibl+ZUJythHoUut55+ixdLNVRDoARL9vix5v8e0XETeRi9zzqvpq9HDathdAVXcD7xPpGi8UEVf0pdrtqWlr9PUCYGeCQz1UJwBjRGQt8CKRWwMPkZ5tbYy0+LltgrT+/GbiNataBly7IAHXr0xIzuYAvaOzKLKAscAbSY4pHt4AxkcfjycyzqH6+CXRGUHHAntqda2nPBER4C/AUlX9U62X0q69ItJWRAqjj3OJjFNZSuRCd160WN22Vv8bnAe8F/2LPOWp6i2q2llVuxP5TL6nqheShm1tpEy5TlVLu89vtUy6ZlXLpGsXJOj6lexBdQkauHcmsJzIPfDbkh1PDNrzd2AzECByX/snRO5fvwusAP4DtI6WFSKzwFYBi4BhyY6/iW09kUj3/0JgQfTrzHRsLzAQ+CLa1q+AO6LHewKfAyuBaUB29HhO9PnK6Os9k92GQ2z3KcBbmdDWg/w7pNV1qla7MuZ6FW1DxlyzarU5I69d0bbE5fplOwQYY4wxxqSQTLitaYwxxhjTYlhyZowxxhiTQiw5M8YYY4xJIZacGWOMMcakEEvOjDHGGGNSiCVnxhhjjDEpxJIzk9JE5C4R+Vuy4zDGmKay65c5VJacGWOMMcakEEvOTMoQkZtEZKOIlIrIMhH5HnArcIGIlInIl9FyBSLyFxHZHC3/WxFxRl+7VEQ+EZFHRWSPiHwtIt9JZruMMenPrl8mllwHL2JM/IlIX2ACcIyqbhKR7oATuBfopaoX1Sr+DJFNg3sBXuAtYD3wRPT1EcA/gDbAucCrItJDVXfFvyXGmExj1y8Ta9ZzZlJFCMgGBoiIW1XXquqquoVEpD2RfeomqqpPVbcBDxLZfLbaNmCSqgZU9SVgGfC9+DfBGJOh7PplYsp6zkxKUNWVIjIRuAs4QkRmADfWU7Qb4AY2i0j1MQeRvzyrbdR9N41dB3SMedDGGINdv0zsWc+ZSRmq+oKqnkjkAqbA/dHvta0HKoE2qloY/WqlqkfUKtNJal35gK7ApnjGbozJbHb9MrFkyZlJCSLSV0ROFZFsoALwA2FgK9BdRBwAqroZeAf4o4i0EhGHiBwuIifXOl074DoRcYvIj4D+wPSENsgYkzHs+mVizZIzkyqygfuAHcAWIheoW4Bp0dd3isj86ONLgCxgCVBCZPBsh1rnmg30jp7rHuA8Vd0Z7wYYYzKWXb9MTMm+t7aNadlE5FLg8ujtBWOMaTHs+mWqWc+ZMcYYY0wKseTMGGOMMSaF2G1NY4wxxpgUYj1nxhhjjDEpxJIzY4wxxpgUYsmZMcYYY0wKseTMGGOMMSaFWHJmjDHGGJNCLDkzxhhjjEkh/w9VWXUZ1rdg3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from nndl import plot\n",
    "\n",
    "plot(runner, 'att-loss-acc.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.3.2 模型评价\n",
    "模型评价加载最好的模型，然后在测试集合上进行评价。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluate on test set, Accuracy: 0.86480\n"
     ]
    }
   ],
   "source": [
    "model_path = \"checkpoint/model_best.pdparams\"\n",
    "runner.load_model(model_path)\n",
    "accuracy, _ =  runner.evaluate(test_loader)\n",
    "print(f\"Evaluate on test set, Accuracy: {accuracy:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.1.4 使用点积注意力模型进行实验\n",
    "\n",
    "对于点积注意力模型，实现方法类似，只需要在双向LSTM的后面加入点积注意力模型，点积注意力模型的输入是双向LSTM的每个时刻的输出，最后接入分类层即可。\n",
    "\n",
    "### 8.1.4.1 模型训练\n",
    "\n",
    "模型训练使用RunnerV3，并使用交叉熵损失函数，并用Adam作为优化器训练Model\\_LSTMAttention网络，其中Model\\_LSTMAttention模型传入了点积注意力模型。然后训练集上训练2个回合，并保存准确率最高的模型作为最佳模型，Model\\_LSTMAttention模型除了使用点积模型外，其他的参数配置完全跟加性模型保持一致。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Train] epoch: 0/2, step: 0/392, loss: 0.69304\n",
      "[Train] epoch: 0/2, step: 10/392, loss: 0.68804\n",
      "[Evaluate]  dev score: 0.49472, dev loss: 0.69163\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.49472\n",
      "[Train] epoch: 0/2, step: 20/392, loss: 0.67802\n",
      "[Evaluate]  dev score: 0.56160, dev loss: 0.67951\n",
      "[Evaluate] best accuracy performence has been updated: 0.49472 --> 0.56160\n",
      "[Train] epoch: 0/2, step: 30/392, loss: 0.64008\n",
      "[Evaluate]  dev score: 0.74208, dev loss: 0.64587\n",
      "[Evaluate] best accuracy performence has been updated: 0.56160 --> 0.74208\n",
      "[Train] epoch: 0/2, step: 40/392, loss: 0.48982\n",
      "[Evaluate]  dev score: 0.77176, dev loss: 0.48411\n",
      "[Evaluate] best accuracy performence has been updated: 0.74208 --> 0.77176\n",
      "[Train] epoch: 0/2, step: 50/392, loss: 0.43586\n",
      "[Evaluate]  dev score: 0.81136, dev loss: 0.42632\n",
      "[Evaluate] best accuracy performence has been updated: 0.77176 --> 0.81136\n",
      "[Train] epoch: 0/2, step: 60/392, loss: 0.36778\n",
      "[Evaluate]  dev score: 0.82968, dev loss: 0.37782\n",
      "[Evaluate] best accuracy performence has been updated: 0.81136 --> 0.82968\n",
      "[Train] epoch: 0/2, step: 70/392, loss: 0.36299\n",
      "[Evaluate]  dev score: 0.83720, dev loss: 0.36994\n",
      "[Evaluate] best accuracy performence has been updated: 0.82968 --> 0.83720\n",
      "[Train] epoch: 0/2, step: 80/392, loss: 0.33715\n",
      "[Evaluate]  dev score: 0.83336, dev loss: 0.38617\n",
      "[Train] epoch: 0/2, step: 90/392, loss: 0.40761\n",
      "[Evaluate]  dev score: 0.84408, dev loss: 0.35747\n",
      "[Evaluate] best accuracy performence has been updated: 0.83720 --> 0.84408\n",
      "[Train] epoch: 0/2, step: 100/392, loss: 0.32692\n",
      "[Evaluate]  dev score: 0.85256, dev loss: 0.34014\n",
      "[Evaluate] best accuracy performence has been updated: 0.84408 --> 0.85256\n",
      "[Train] epoch: 0/2, step: 110/392, loss: 0.42844\n",
      "[Evaluate]  dev score: 0.85688, dev loss: 0.32720\n",
      "[Evaluate] best accuracy performence has been updated: 0.85256 --> 0.85688\n",
      "[Train] epoch: 0/2, step: 120/392, loss: 0.30796\n",
      "[Evaluate]  dev score: 0.86032, dev loss: 0.32698\n",
      "[Evaluate] best accuracy performence has been updated: 0.85688 --> 0.86032\n",
      "[Train] epoch: 0/2, step: 130/392, loss: 0.44230\n",
      "[Evaluate]  dev score: 0.84176, dev loss: 0.36484\n",
      "[Train] epoch: 0/2, step: 140/392, loss: 0.32406\n",
      "[Evaluate]  dev score: 0.85168, dev loss: 0.33859\n",
      "[Train] epoch: 0/2, step: 150/392, loss: 0.29786\n",
      "[Evaluate]  dev score: 0.84160, dev loss: 0.35914\n",
      "[Train] epoch: 0/2, step: 160/392, loss: 0.23639\n",
      "[Evaluate]  dev score: 0.84624, dev loss: 0.35123\n",
      "[Train] epoch: 0/2, step: 170/392, loss: 0.31961\n",
      "[Evaluate]  dev score: 0.86504, dev loss: 0.31720\n",
      "[Evaluate] best accuracy performence has been updated: 0.86032 --> 0.86504\n",
      "[Train] epoch: 0/2, step: 180/392, loss: 0.20162\n",
      "[Evaluate]  dev score: 0.85024, dev loss: 0.34308\n",
      "[Train] epoch: 0/2, step: 190/392, loss: 0.24377\n",
      "[Evaluate]  dev score: 0.84288, dev loss: 0.35614\n",
      "[Train] epoch: 1/2, step: 200/392, loss: 0.23590\n",
      "[Evaluate]  dev score: 0.86984, dev loss: 0.30419\n",
      "[Evaluate] best accuracy performence has been updated: 0.86504 --> 0.86984\n",
      "[Train] epoch: 1/2, step: 210/392, loss: 0.23255\n",
      "[Evaluate]  dev score: 0.85816, dev loss: 0.36808\n",
      "[Train] epoch: 1/2, step: 220/392, loss: 0.20135\n",
      "[Evaluate]  dev score: 0.85392, dev loss: 0.34667\n",
      "[Train] epoch: 1/2, step: 230/392, loss: 0.17271\n",
      "[Evaluate]  dev score: 0.85512, dev loss: 0.36192\n",
      "[Train] epoch: 1/2, step: 240/392, loss: 0.11344\n",
      "[Evaluate]  dev score: 0.85432, dev loss: 0.42230\n",
      "[Train] epoch: 1/2, step: 250/392, loss: 0.12108\n",
      "[Evaluate]  dev score: 0.86080, dev loss: 0.33558\n",
      "[Train] epoch: 1/2, step: 260/392, loss: 0.05440\n",
      "[Evaluate]  dev score: 0.85952, dev loss: 0.38455\n",
      "[Train] epoch: 1/2, step: 270/392, loss: 0.14359\n",
      "[Evaluate]  dev score: 0.85880, dev loss: 0.36819\n",
      "[Train] epoch: 1/2, step: 280/392, loss: 0.04112\n",
      "[Evaluate]  dev score: 0.85952, dev loss: 0.39924\n",
      "[Train] epoch: 1/2, step: 290/392, loss: 0.13611\n",
      "[Evaluate]  dev score: 0.85376, dev loss: 0.41588\n",
      "[Train] epoch: 1/2, step: 300/392, loss: 0.05073\n",
      "[Evaluate]  dev score: 0.85488, dev loss: 0.38210\n",
      "[Train] epoch: 1/2, step: 310/392, loss: 0.14445\n",
      "[Evaluate]  dev score: 0.85504, dev loss: 0.40037\n",
      "[Train] epoch: 1/2, step: 320/392, loss: 0.06760\n",
      "[Evaluate]  dev score: 0.85632, dev loss: 0.37916\n",
      "[Train] epoch: 1/2, step: 330/392, loss: 0.10616\n",
      "[Evaluate]  dev score: 0.84096, dev loss: 0.49569\n",
      "[Train] epoch: 1/2, step: 340/392, loss: 0.03065\n",
      "[Evaluate]  dev score: 0.85304, dev loss: 0.43344\n",
      "[Train] epoch: 1/2, step: 350/392, loss: 0.05785\n",
      "[Evaluate]  dev score: 0.84744, dev loss: 0.48169\n",
      "[Train] epoch: 1/2, step: 360/392, loss: 0.13306\n",
      "[Evaluate]  dev score: 0.84992, dev loss: 0.44447\n",
      "[Train] epoch: 1/2, step: 370/392, loss: 0.12645\n",
      "[Evaluate]  dev score: 0.84768, dev loss: 0.42809\n",
      "[Train] epoch: 1/2, step: 380/392, loss: 0.12673\n",
      "[Evaluate]  dev score: 0.84864, dev loss: 0.42662\n",
      "[Train] epoch: 1/2, step: 390/392, loss: 0.10942\n",
      "[Evaluate]  dev score: 0.84992, dev loss: 0.40914\n",
      "[Evaluate]  dev score: 0.84136, dev loss: 0.43722\n",
      "[Train] Training done!\n",
      "训练时间:98.72966623306274\n"
     ]
    }
   ],
   "source": [
    "paddle.seed(2021)\n",
    "# 实例化基于LSTM的点积注意力模型\n",
    "model_atten = Model_LSTMAttention(\n",
    "    hidden_size,\n",
    "    embedding_size,\n",
    "    vocab_size,\n",
    "    n_classes=n_classes,\n",
    "    n_layers=n_layers,\n",
    "    use_additive=False,\n",
    ")\n",
    "# 定义优化器\n",
    "optimizer = Adam(parameters=model_atten.parameters(), learning_rate=learning_rate)\n",
    "# 实例化RunnerV3\n",
    "runner = RunnerV3(model_atten, optimizer, criterion, metric)\n",
    "save_path = \"./checkpoint/model_best.pdparams\"\n",
    "\n",
    "start_time = time.time()\n",
    "# 训练\n",
    "runner.train(\n",
    "    train_loader,\n",
    "    dev_loader,\n",
    "    num_epochs=epochs,\n",
    "    log_steps=10,\n",
    "    eval_steps=10,\n",
    "    save_path=save_path,\n",
    ")\n",
    "end_time = time.time()\n",
    "print(\"训练时间:{}\".format(end_time-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "```\n",
    "训练时间:148.5727310180664\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化观察训练集与验证集的损失及准确率变化情况。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAFDCAYAAAB/Z6msAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4XOWV+PHve6ePuizJkuVuSy5yA9zABJsspmRpIYE1SxJIICzskmQJv2wIm7bJpjcIIbubzW5INrsQElJIQkkooTeb5iL3LluWZFtWm37P7487kmWrWGU0M5LP53n0WLrz3jtHxXfOvOW8RkRQSimllFLZwcp0AEoppZRS6jhNzpRSSimlsogmZ0oppZRSWUSTM6WUUkqpLKLJmVJKKaVUFtHkTCmllFIqi2hyppRSSimVRTQ5U0oppZTKIpqcKaWUUkplEXemAxiOkpISmTp1aqbDUEql0bp165pEpDTTcQyX3r+UOv0M9P41qpOzqVOnsnbt2kyHoZRKI2PMnkzHkAp6/1Lq9DPQ+5cOayqllFJKZRFNzpRSSimlsogmZ0oppZRSWUSTM6WUUkqpLJK25MwYc7ExZosxZrsx5s5eHv+eMeat5MdWY0xzumJTSimllMoWaVmtaYxxAfcBq4H9wOvGmEdEZFNnGxG5vVv7jwFnpCM2Nfq0tLTQ0NBALBbLdCgqxTweD2VlZeTn52c6FKWUyph0ldJYCmwXkZ0AxpgHgSuATX20vxb4QppiU6NIS0sLhw4dorKykkAggDEm0yGpFBERQqEQdXV1AJqgKaVOW+ka1qwE9nX7en/yWA/GmCnANODpNMSlRpmGhgYqKysJBoOamI0xxhiCwSCVlZU0NDRkOhyllMqYbFwQsAb4lYgkenvQGHOzMWatMWZtY2NjmkNTmRaLxQgEApkOQ42gQCCgQ9ZKqdNaupKzOmBSt68nJo/1Zg3wQF8XEpEfichiEVlcWjqwHVzEtmn97kvYrZGBxquymPaYjW36+1VjWeJYB3ZrONNhqCyXruTsdaDKGDPNGOPFScAeObmRMWY2UAS8nMonj71xkJZ/+jNH/uaXSPzEDrl4YwvxuqOpfDqllFKqBxGh44kNtP/hLew27SxQfUtLciYiceA24AmgFnhIRDYaY75kjLm8W9M1wIMiIql8fu/iSgp/+NdEHtvOsY89RvfLR9/aS8efNhB5ey8pflqllFKqi7SFIZ5AwjE6nqlFEnamQ1JZKm1zzkTkURGpFpEZIvKV5LHPi8gj3dp8UUR61EBLhZybF5N757m0//ta2r71YtfxwKo5uKeXEnljD6Gna5FofCSeXqkRtXnzZowxw95I+/HHH8cYQ1NTU4oiU2pssENRxB7eG3grL0DummUEVs7Gbmol/OqOFEWnxppsXBAwYvK/8m4Cf1NDy6efpOOhDQAYj4vAebPwLZ1OfN9hp7u5I5rhSNVYY4zp92Pq1KnDun5VVRUHDx5k0aJFqQlYKdXFDsdo+/VaQs9uHvI1RAQRwVgWnuml+M6YgntCUQqjVGNJuuqcZQVjWRTdfyWJ/S0c/dBvcFXm41sxGWMMvppKXONyiG09hPF7Mh2qGmMOHjzY9flLL73E+973Pt544w0qKioAcLlcvZ4XjUbxer2nvL7L5aK8vDw1wSqlThDdWAfRBPHdTcT2HcEzqXjQ10gcPEboxW0EL5iLqygH36LJXY9Jwsa4Tqu+EnUKp91fg/F7KP7dtbgmF3DkigeIbzvc9Zi7vJDAebMwlkEicex2nbCpUqO8vLzro7jYubGXlpZ2HetceVxeXs6//Mu/cPPNN1NcXMzq1asB+Pa3v82CBQvIyclhwoQJfOADHzihFtjJw5qdX//617/mkksuIRgMMnPmTP7v//5v0LG/8MILnHvuufj9foqLi/nQhz7E4cPH/9/s2bOHK6+8knHjxhEIBJg5cyb33HNP1+O/+tWvWLhwIcFgkKKiIs4++2w2bNgw+B+iUhkgkRjR2gO4J4/Du3AS7rK8IV0ntqsRCUex8vwnHI9uP0Tbb9Zhh0e2fEz3IVmJxIY9RKtG1mnVc9bJNS5IyaPX0Xj2f9F0yc8pffkmXKU5XY+LLbT9dh2u8QUEV83OYKRqIJr/8TFib9Wn/Xk9i8opvPuSlF/3O9/5DnfeeSevvvoqiYSzutgYw9133820adM4cOAAt99+Ox/84Ad54okn+r3Wpz/9ab7xjW9w77338u///u/ccMMNnHPOOQMeRt23bx8XXXQRV199Nf/xH/9BU1MTt9xyC2vWrOHPf/4zAB/96EdxuVw8/fTTFBQUsGPHjq7kbe/evaxZs4bvfOc7XH755YRCIdatW9dnT6FSvZFIjNiuRjwzx2Pc6f3bkZiNe0IhvkVTcBU7rxMiMqiSL2LbxPc04Z40rkf8rsIg0hEh9JfNBC+ch7FSW0pGRIhtqSe6qY6cv16E8bnp+MtmEgePYeX5MXl+rFwfrqIcvHMmAJBobAXLgNtCInEkHMN43bjLCwCIvLkHz6wKrOCpe/XV0JyWyRmAe+Y4ih+5lqbz7+fIFQ9Q8tT1mIAznGksg2daKdFNdditU3u801FqJL3rXe/irrvuOuHYHXfc0fX5tGnTuOeeezjnnHM4fPgw48aN6/Nat99+O1dddRUAX/3qV7n33nt59tlnB5ycff/732f8+PH8+Mc/xu12bhf3338/y5cv57XXXmPp0qXs2bOHD3/4wyxcuBDghGvX1dVh2zbXXHNN1xDu3LlzB/Tc6vQmthCvO0ps+yHiew+DLZiAFxPwEtt+CP/ZM9NSE8/K9RF89/G/2cSxDkLPbSWwoqorWTuVxMFjSCSOZ1pJj8dcJXn4l88k/OI2wi9sxXfmVKxcX0pit0NRwi9uI77vCK4JhYhtYwDvrHISJXnYLSHs1jDxplbsllBXchZ6fgv2sdAJ13JPKsZdXoCIEN1ykMimOvxnTcMzq1xrE46A0zY5A/CdPYnin1/Fkat/ydHrf0PRg+/HWM5Ir3duJdFNB4huqsO/bEaGI1X9GYneq0xaunRpj2NPPvkk3/jGN9i8eTPNzc3YtrMEf8+ePf0mZ90XCHi9XkpKSjh06NCAY9m4cSPnnHNOV2LWGZ/f72fjxo0sXbqUT37yk9x222387ne/Y9WqVVx66aWsWLECgCVLlrBy5UpmzZrF6tWrWbVqFVdddRWVlb3u3qYUAHZ7hPbfv4mEYhifG+/sCjwzx+Mal0t02yFiW+pxTyzGM7nvv/1UiO09jJUfwFUY7DpmfB6kNUT45e0E37NgQIlJbFcjeFy4K3ufq+atLsduDRNdv4/Yniby1izDeIb38hzbd4TwC1uRWBzfsul450zoitUztRTP1BOLuHcvJeV/VzXSEUXiNsbnxvJ5MMleMmMMwUsWEH5pO+GXtxPb0YD/nJm4igaWqPbHDkVJHGjGDkXxVpdjvKdvinLazTk7WeD9NeR/azWhX26i5c4nu45buT4800uJbq1HIrqVjEqfnJwTb3Lbt2/n0ksvZdasWfziF79g7dq1/PKXvwScBQP9OXkxgTGmK7FLlb/7u79j165d3Hjjjezdu5fVq1dz0003AeB2u3n66af505/+xBlnnMGDDz5IVVVV15CoUp3sjiixvc5wuAl68UwvI/DuueT+zTL8y2bgGpcLgGdGGSbPT+Stka1NKbEE4Re2Enl91wnHLb8H35LpJBpaiG0d2HQK99QS/GdNxbj7fsn1nzWV3PcvIbCiqisxC6/bTby+GRHBbo8Qrz/WVU0g0dRKxzO1hF7eTuTNPUQ3HyC2u8kp+SFCdFMdJuAl57Iz8M2tPGUS2f1xd2k+nikleGeU4ZlYjKs0DyvneG+eqyBI8OL5+M+txj7WQfsf3z6hDJVE4iQOtxHbc5jIpjoSTa2A8zsOvbiN0AtbCT23hY5namn7/Ztdv3e7uYPQc1uIvL6L8NoTf+6nm9M3Le0m945zSOxqpu1bL+GaVkTurUsA8M6rJLajgfj+o3hmlGU4SnW6evXVV4nFYtx9991dPVgvvvjiKc5KjZqaGh5++GHi8XjXc7/22muEw2HmzZvX1W7ixIncdNNN3HTTTdx///185CMf4b777sPn82GMYfny5SxfvpzPfvazrFq1ivvvv79rsUM2McZcDNwDuIAfi8jXT3p8MvBToDDZ5k4RedQYMxWnwPaWZNNXROSWdMU9mtktISLr9xPbfghcFu6/WYbxuPAvnd5re2MZfAsmOcN1+48OaeXkQEQ3H0QicbwLJ/V4zDOzjNj2Q4TX7sY9eRxWoP+5V56JA4vRyvVj5TrTaOxQlNiWg0Tf2efM/0pO4PevqMJbXQ4uC/toOxKOIZHjiVFgdQ2eicUEVs7GeFwjtgrUGIO3ajzuSUUkGloxXjd2KErbw2shdtLW2GdOwVWSB/EE8X2HwRiwDMZlYXJ8XTG6SvLIee+ZxDbXE609gLe63DlvAMSWlM/XyyRNznD+yAruuZj4nmaO3fYo7ikF+N9Tjas4l9z3LcbK1422VeZUV1dj2zbf+973eP/7388bb7zB1772tbQ89yc+8Ql++MMfctNNN/GpT32KpqYmbr31Vi644AKWLHHexNxyyy1ceeWVVFVVEQqF+O1vf8uMGTPw+Xz85S9/4aWXXuKCCy6gvLyczZs3s2nTpmxNzFzAfcBqYD/wujHmERHZ1K3ZZ3F2OPk3Y8xc4FFgavKxHSKiheYGIbrpAOHXdoAxeKrG45s3EeM59YR/z8wyIm/vJfLWHtwTi/rsFRIREAb9oi3xBNGN+3FNKMRdlt/jcWMM/rNn0v67N4huqMO/ZFqf14rtacIqDOIqCPbZpjdWwEvuNUuJbW/Abg1j5fmw8gJYyXlurqIccq9a7MRr20g4joSOrwa10lQSyvJ7sZLDy8bnwTtzPCbXh5Xrw8r1Y3J9GJ8Ti5UfIG/N8j6vZTwuXIU5WGdOIba7kfArOwj+9cJT9vrZLSHan1iP/8ypuKeXjok5cKf9sGYn43ZR/OD78Swcz5Frfkn0jQMAXYmZbrOhMmXJkiV897vf5Z577mHu3Lnce++9fO9730vLc0+cOJEnnniCbdu2cdZZZ/He976XxYsX8+CDD3a1SSQSfOxjH2PevHmsXLmSRCLB73//ewCKiop47rnnuOyyy6iqquLmm2/mxhtv5NOf/nRa4h+kpcB2EdkpIlHgQeCKk9oI0PlqXQAcSGN8Y4rdEiL8+k7cE4vJvXopgXOqBvxG2FgW/rOmOvOm+hjZFBEir+8i9OzmQZeNiG6pR0IxfAsn99nGVRgkeNE8fGdO6bONxBOEnttKdEPdoJ6/k3G78M6uwL9kGt7ZE3BXFvXaS2csCyvoxTUuN6PztIxl8C+fgW/eRDxTS3GV5GH5vYNOlozXjX/ZDNzT+v79wvFdG8S2MX6PM1T6+HoSzR19niO2TaKx9YTexmxkRvN+kosXL5bhbldzssSBFhqX/xiJ25S+chPuyYWE39xDfM9hcq44Y0xk5KNZbW0tc+bMyXQYaoT193s2xqwTkcWpfk5jzPuBi0XkpuTXHwSWicht3dpUAH8CioAc4AIRWZcc1twIbAVagM+KyPO9PMfNwM0AkydPPmvPnj2p/jZGlXhDC66SvBEZjoq8vZfIG3vwzpmAb5kzR8xVkjegYb7IW3uJN7SQc+G8U7YFsNvCxOuP4Z05/oTjsd1NhJ6pJXjRPN0NIMUknqD9j29j5QcInj8HsYXY1nrC63ZDPIF3/kR8ZziJs90SInGgmfiBZuIHmyGWIOe9Z+EqDDrz+I514CovwMoPYIw5oVRK/NAx7OYO7Naw89EScubyXTiPxOE2JJboKjEyEAO9f+mw5klcE/IZ9+h1NK74bw7/9f9R+sJHcOUHiB5tH9H5DUqpUeFa4H4R+Y4x5mzgf4wx84CDwGQROWyMOQv4rTGmRkRaup8sIj8CfgTOm8t0B59pkrAJv7AV9+RxeKaV9jpkOKjr2UJsZwNWnh/3+OMvkNHNB4i8sQfPjDJ8y6YjrWE6HluPe1qJU2j8FG+yfYsm4x1Ex0V0Qx3R2gMkmlrxL53eteo/tqsR4/fgKi8c2jeoiO5owD7Wgf/MqSccD7+6E/tIO76znOPGMnhnV+CeMo7I2l0QS2CMIbb/CKE/b3Ta5Pqcv7uKAqwCp5c2truRWK2zg4sJeMBlAYa8q51pG9H1+4nvOwKWceYE5vmxSpzFKZE3dmOHYuRefkbKv29NznrhmTee4oev4fAl/8uR9z9E8R+uxeT4iG7Yr8mZUmNXHdB99vfE5LHubgQuBhCRl40xfqBERBqASPL4OmPMDqAaSG3X/igmsQQdz9SSqDuKVZyboosKkXW7sXL9uJJlLWI7Gwm/vAP3pGL851Y5+9fmB/CdOYXIut1E/B58S6f3SNBExKmnZln9zmPrjW/pdLAM0Y112EfaCZw/B+N2Ed9/xCmcO4Ymqqeb3dhKtPYAnskluJJJUXT7IWJb6/EumNRjsYUV8BJ416yulbzusnz8Z8/APaEIk+fv8Xv1L5uBd84EEvXHSBxy3kuZbitT/ctmwPKZmKC3x+/RBH1IU1vKv2fQOWd98l8wg8L/vIzIkzs5dusf8c5N/vIaWzMdmlJqZLwOVBljphljvMAa4JGT2uwF/grAGDMH8AONxpjS5IICjDHTgSpgZ9oiz3ISidPxpw0kDhzFf85MfPMnpuS6xmXhWzCJREMLiYPNzrEcL+5JxQRWze7qwQLwzp+Id+4Ep37l+v3HYxMhtvcw7Y+8SejpWiKv7sBuCfV4rn7jsAz+pdMJnDeLRFMb7Y+8SWxnA9iCZ1rpqS+g+uQ7YwrG7yH8ynZEhMTRdsIvbcdVXtA1bNmbziTMeN14Z0/oGrLsrZ2rIIh3VgWB82YROG8W/mRvHOD0lOX6ek2wrRyvs1p2BOaka89ZP3JuOIPErmZav/QsrhlFMNVDZON+gqt0zpNSY42IxI0xtwFP4JTJ+G8R2WiM+RKwVkQeAe4A/tMYczvOVOUbRESMMecBXzLGxAAbuEVEjmToW8kqYtt0PL2JRFMrgVVz8EztWSV/ODzV5UTe2Uf4lR3kvPcs3OMLThji7GSMwbd0OnY4RmTdblzj8zEui9BL27EPt2Hy/PjPrXbqqA2xp8szowyrMOgMqU4vcxYseHWrsuEwPje+xdMIv7CV2PZDuIpycBXnOKVCMtwjaYJOD5t0RDEp3klIk7NTyPviKuK7jtL62Wco+Oml+Bf3XntHKTX6icijOOUxuh/7fLfPNwErejnvYeDhEQ9wNDIG1/h8PFXjU56YQbL3bP4kwq/uILphP775PeuSHQ/FEDi3mlhlEa6yfOyWMMQS+M+twjMjNcOPrnG5BFfXDPs66jjPzDJiW+uJrN1N7vsXD6i8Rjp0Fua1OyIp3+ZRk7NTMMZQ9OPLSexr4dhNj+KZPA7fqr5r2iillHJILI7xuHtM5k41z6xyJJEY0BCicVldqypdBQFyrjorK17oVd8668rZbWFwu7Lm9+Uan0/u1Uu6etBSSeecDYDxuhn367/BPb+UY996tmsrCqWUUr2L7Wqk7Vdr+605lSqdvWed1fUHdW6WvNCr/rmKc/BMHpdVvy/jdjmFdkdgeFV7zgbIKgqQ+//OJhFuJbq+nsD5A9tSQimlxhIRccoL7D2Me2oJ3qrxXRXgO8UbWgg9v8UpQjqEhEmp0532nA2CZ7bTZZ7Y15zhSJRSKjOkNUzkrb3OxPrXd9H6i9cIvbAVO+xsyG23hgg9uRET9BF499x+N/tWSvVOe84GwT23DFm3ncSRkalrotRIu+GGG9i/fz9PPvlkpkNRo4xE4hifGys/QM7lZ2AVBLCPdhDdfID4/qP43S4kYRN6fisikLO6Jm37Oyo11uhbmkGwfB7oSCCtkUyHokahG264wSmIaQwej4eSkhLOPfdcvvnNb9Le3p7p8JTqU/zQMdp+vZbotnrA2VfSGOOUNDinitz3L8G4XWAZJBon+FdzBr3Rt1LqOE3OBst2ViApNRTvete7OHjwIHv27OGZZ57huuuu4wc/+AFnnnkmhw4dynR4SvUQ3VpPx+PrMV43rtLet1vqPiE65z0Lcet2RUoNiyZng2TaLDq+/xZ2RzTToahRyOv1Ul5ezoQJE5g/fz633norL7/8Mo2Njdx5550ntL333nuZPXs2fr+fqqoqvvKVrxCPO28M/vmf/5lZs2b1uP6tt97KueeeO+B4RIRvf/vbTJ8+Ha/Xy4wZM7j77rtPaPO73/2OM844g2AwSGFhIUuXLuXNN98EIBaL8clPfpKJEyfi8/moqKhgzZo1g/2xqCwVrT1A+MVtuCoKybl0Ia7C/nvDjDEYr86WUWq49H/RIHlml0JbnHhtE96zJmQ6HJXU/tg7PY55ppbgnTMBiSfoSG58e8LjM8fjrRqPHY4Reqa2x+PeWRV4ppdit0UIPb+lx+M5lyxISeyVlZVcd911/OxnP+O//uu/sCyLL37xi/zkJz/h7rvvZtGiRdTW1nLLLbcQDof58pe/zPXXX89Xv/pVXn31VZYtWwZAJBLhF7/4BV//+tcH/Nw//OEP+dznPsc999zD+eefz1NPPcU//uM/kpeXx4033kh9fT1XX301//qv/8rVV19NOBzmzTffxO12bh333nsvDz30ED//+c+ZPn06hw4d4sUXX0zJz0VllsQSRN7cg6uikOAFNRmvxq7U6USTs0GypuThvXI6sQ2HNDlTKVNTU0NLSwtNTU3k5ubyzW9+k1//+tdcfPHFAEybNo1//dd/5eMf/zhf/vKXqa6uZtmyZfzsZz/rSs5+//vfEwqFuOaaawb8vF//+tf52Mc+xs033wxAVVUVW7Zs4Stf+Qo33ngjBw8eJBaLcc011zB16lQA5sw5vn3Znj17qK6uZuXKlRhjmDx5MkuWLEnRT0VllNvCf94srEDPDZ+VUiNLk7PBClj4Lp9GfEtTpiNR3fTXi2Xcrn4ft/ye/h/P9aWsl6wvIgI4w0IbN24kFArxvve974SCi4lEgnA4TGNjI6WlpVx//fV87nOf4+6778bj8fCzn/2Myy+/nMLCgc33aWlpYf/+/Zx33nknHF+5ciX33HMPHR0dLFiwgIsuuoh58+axevVqVq1axVVXXcWkSc4WOR/+8IdZvXo1M2fOZPXq1axevZrLLrsMr9ebop+MyhRjDJ6JxZkOQ6nTks45G6TOgorx3UczHIkaSzZu3EhBQQHjxo3Dtm0AfvnLX/LWW291faxfv55t27ZRXOy8YK5Zs4bW1lb++Mc/0tjYyOOPP87111+f0rhcLhePPfYYTz/9NEuWLOHhhx+murqaP/zhDwAsWrSIXbt28e1vfxuv18snPvEJFi1aREtLS0rjUOkV2VRHeO0uxJZMh6LUaUmTs0EyAaduT+KQbuGkUqOuro7//d//5aqrrsKyLGpqavD7/ezcuZOZM2f2+HC5XAAUFRVx2WWX8T//8z888MADFBcXc9FFFw34efPz85k4cSLPPffcCcefffZZpk2bRjDoTP42xrB06VLuuusunnvuOVauXMlPfvKTrva5ubm8973v5fvf/z5r166ltraWZ599NgU/GZUJEosTfWsv9pF2Hc5UKkN0WHOQrIAzXCOROHZbBCs39RueqrErGo1SX1+PbdscPnyYF154ga997WuUlZXxta99DXCSnbvuuou77roLYwwXXHAB8Xic9evX8+abb/KNb3yj63of+tCHuPrqq6mtreW6667rStwG6jOf+Qx33HEHVVVVrFq1iqeffpp/+7d/47777gPgpZde4qmnnuLCCy+koqKCbdu28c4773DjjTcC8K1vfYsJEyawaNEigsEgDzzwAC6Xi+rq6hT9xFS6RWsPIpE43jMmZzoUpU5baUvOjDEXA/cALuDHItJjSZkx5hrgi4AAb4vI36YrvoEyyeTMKvAS39SId+nEDEekRpPnn3+eiooKXC4XBQUFzJkzh9tuu41/+Id/ICcnp6vd5z73OSoqKvjBD37AHXfcQSAQoLq6mhtuuOGE611yySUUFBRQW1vLAw88MOh4br31Vtrb2/nqV7/K3//93zNp0iS+/vWvdyVfBQUFvPzyy9x3330cPXqU8vJyrrvuOj73uc8BTu/bd7/7XbZt24Zt28yZM4eHH3641zIfKvtJLE50w37clUW4+6hpppQaeaZzIvKIPokxLmArsBrYD7wOXCsim7q1qQIeAt4tIkeNMWUi0tDfdRcvXixr164dwch7F6ttoGHuDyn8r8vJ+ciZaX/+01ltbe0JqwXV2NTf79kYs05EFqc5pJTL1P2rP5F39hFZt5ucSxfhKs3LdDhKjTkDvX+lq+dsKbBdRHYCGGMeBK4ANnVr81HgPhE5CnCqxCyT3NUl4HcT39iY6VCUUipl3MnVmZqYKZVZ6VoQUAns6/b1/uSx7qqBamPMi8aYV5LDoD0YY242xqw1xqxtbMxMchTb0YD/A7OJbcza/FEppQbNVZyDb8GkTIeh1Gkvm1ZruoEqYBVwLfCfxpgeBZtE5EcislhEFpeWlqY5REe87ijuheO050wpNSZINE7oxW3YraFMh6KUIn3JWR3Q/e3YxOSx7vYDj4hITER24cxRq0pTfINiBbyYgJvE/hbsZr2ZKaVGt+iWemJb65FwPNOhKKVIX3L2OlBljJlmjPECa4BHTmrzW5xeM4wxJTjDnDvTFN+gmKDX+cl5LWKbtPcs3dKxiEVljv5+0y++/whWcY7ONVMqS6QlOROROHAb8ARQCzwkIhuNMV8yxlyebPYEcNgYswl4BviUiBxOR3yD1VlOwxR4dWgzzTweD6GQ9laOZaFQCI/Hk+kwThti2ySaWnGPL8h0KEqppLTVORORR4FHTzr2+W6fC/DJ5EdWswIecFlY4wK6KCDNysrKqKuro7KykkAgcMLek2p0ExFCoRB1dXWMHz8+0+GcNuzD7RC3cY3XumZKZQvdIWAIXJVF5H3wHML3biS+QZOzdMrPd15ADhw4QCwWy3A0KtU8Hg/jx4/v+j2rkSeRGCbXh6tMf+ZKZQtNzoags7fGU1NK+IkdGY7P05soAAAgAElEQVTm9JOfn68v3mpEnGonE2PMZOCnQGGyzZ3JUQGMMZ8BbgQSwMdF5Il0xj5U7onF5F29NNNhKKW6yaZSGqOGiBB6fiuuM0ux69uwj3RkOiSl1DAldzK5D7gEmAtca4yZe1Kzz+LMmT0DZ2HTD5Pnzk1+XQNcDPwweb2sJiK6AEOpLKTJ2RAYY4jXHcEq8wMQ00UBSo0FXTuZiEgU6NzJpDsBOrttC4ADyc+vAB4UkUiyFND25PWymrSFafvFa8TrjmY6FKVUN5qcDZEJeCHojArHdVGAUmPBQHYy+SLwAWPMfpwFTh8bxLlZscNJd/FDLUgo2rUCXSmVHTQ5GyIr4AUEk+clposClDpdXAvcLyITgfcA/2OMGfB9NBt2OOkucagFvC6somCmQ1FKdaPJ2RCZoBcJRXHPLdVhTaXGhoHsZHIj8BCAiLwM+IGSAZ6bdRINLbhL87UkjVJZRpOzIbJy/RivG8+8Mh3WVGpsGMhOJnuBvwIwxszBSc4ak+3WGGN8xphpOFvPvZa2yIdAIjHs5g6tb6ZUFtLkbIh8iyaT+96z8NSUYTd2kGhoy3RISqlhGOBOJncAHzXGvA08ANwgjo04PWqbgMeBfxCRRPq/i4GThOCdMwF3ZVGmQ1FKnUTrnA2Tu8aZNxLf2IirLDfD0SilhmMAO5lsAlb0ce5XgK+MaIApZAW9+JfPyHQYSqleaM/ZECWOddD+2DtYlTkAuo2TUmpUSTR3ILbWOFMqG2lyNmSGRP0x8FmYQr9ugK6UGjUkYdP+yBtE1u3KdChKqV5ocjZEVtADgISieGpKtedMKTVqJA63QUJwlepiAKWykSZnQ2Q8bnBbSCiGu6aM2IYG3QZFKTUqJA61AOhKTaWylCZnw2ACXuxkz5kcDWPX64pNpVT2SzS0YOX5k8W0lVLZRpOzYXCX5WMFvHjmlQG6KEAplf1EhERDi/aaKZXFtJTGMATOmwVA4pDTYxbf2AgX6NJ0pVR2C6ychfHq7V+pbKX/O1PAKsvBGhfQPTaVUlnPGIN7ghaeVSqb6bDmMES3H6Lt4bWQENw1uo2TUir7xfY0Ea8/lukwlFL90ORsOGzBbgkh4c5yGo26YlMpldUia3cT3bA/02EopfqhydkwmORKJ7sjinteGdISwa5ryXBUSinVOzscxW4J4SrTxQBKZTNNzoahcxm6U4i2c8Wm7hSglMpOWt9MqdFBk7NhMMHO5CzWtQG6LgpQSmWrREMLWAbXuLxMh6KU6ocmZ8Ng/B5cEwqdf0tysMpydFGAUiprJRpacZXlY9x661cqm2kpjWEwliHnovldX3vmlemwplIqawUvWYCEY5kOQyl1Cvr2KYXcNaXENzUitp3pUJRSqgdjGaygbtmkVLbT5GyYQs9vof2xdwDw1JQhbVESe7WGkFIqu4Se20JkY12mw1BKDYAmZylgt4YBuhYFxHVoUymVRexwlNjOBiQaz3QoSqkB0ORsmEzAi4SiiEi3chq6KEAplT3i+46AgGfyuEyHopQagLQlZ8aYi40xW4wx240xd/by+A3GmEZjzFvJj5vSFdtwmIAXbIFoHKsogDUhTxcFKKWySnzvEUyOD6s4J9OhKKUGIC2rNY0xLuA+YDWwH3jdGPOIiGw6qekvROS2dMSUKp2FaO1QFJfPg6emlLjWOlNKZQmJJ4jXHcVTPR5jTKbDUUoNQLp6zpYC20Vkp4hEgQeBK9L03CPKKgziqRqPsZwfpbumjHitrthUSmUHiSbwTBmHZ2pppkNRSg1QupKzSmBft6/3J4+d7H3GmHeMMb8yxkxKT2jD4yrOIXBuNVZ+AHBqnUkoTmJXc4YjU0opsIJeAitn4y4vyHQoSqkByqYFAb8HporIAuDPwE97a2SMudkYs9YYs7axMTvmdokIknB6yjyd2zjpogCVhez2CPG6o5kOQ6WJ2EKiuQMRyXQoSqlBSFdyVgd07wmbmDzWRUQOi0gk+eWPgbN6u5CI/EhEFovI4tLS7Oimb/2/V4i8sRsA99xkOQ2dd6ayUPi1nXQ8uTHTYWStASxc+l63RUtbjTHN3R5LdHvskfRG3rtEQwvtv1nnrNZUSo0a6dq+6XWgyhgzDScpWwP8bfcGxpgKETmY/PJyoDZNsQ2b8bqRjigAVr4f16R8XbGpslMsgVUYzHQUWWkgC5dE5PZu7T8GnNHtEiERWZSueAcivvcwWEaHNJUaZdLScyYiceA24AmcpOshEdlojPmSMebyZLOPG2M2GmPeBj4O3JCO2FLBCniwQ8f3q3PXlOkG6Cor2W1hjMdF4mh7pkPJRoNduHQt8EBaIhsCESG+9zCuikKMV7dRVmo0SducMxF5VESqRWSGiHwleezzIvJI8vPPiEiNiCwUkfNFZHO6YhuuzkK0nTzzyohtbuqah6ZUNhAR7LYwiUMthF/clulwstFAFy5hjJkCTAOe7nbYn5wP+4ox5so+zkvbnFn7WAd2a1gLzyo1CmXTgoBRywS92B2Rrkm37ppSiCSI79B5Hip7SEcUEgJeN4mmNiSWOPU5tiC2TibvxRrgVyLS/Yc4RUQW40zZuNsYM+Pkk9I5Zza+5zAA7snFI/o8SqnU0+QsBTyTivHNm+i88EHXNk66KEBlFWPwzpuIb/5EECHR0HLKUzr+tIGOR99OQ3BZ4ZQLl7pZw0lDmiJSl/x3J/AXTpyPlnbeORMIrK7BCvoyGYZSagg0OUsB98RifAsnY9zJQrRzSgB0UYDKKlbQi3/JNLxzKsBAvP5Yv+3tcIzEwWYSja2nyxy1roVLxhgvTgLWY9WlMWY2UAS83O1YkTHGl/y8BFgBnLwDSloZrxvPRO01U2o00uQsRSQaJ97o9ERYuT5c0wp1UYDKKnY4hsQSGI8ba1weiUP9J2fGbeFf7ozMRbcc7LftWDDAhUvgJG0PyonFw+YAa5MLmp4Bvt7L9nRpE9t7mMj6/bpTiVKjlC7hSZHwut3EtjeQd91yjGXhqSnTnjOVVSKv7SRef4y8a5YSWD4D4+v/v79xu/DOmUCisZXYjgb8i6d39Q6PVSLyKPDoScc+f9LXX+zlvJeA+SMa3CDEdjSQaGrFO6/X9QxKqSw3tu+0aeSuKIR4gkRjq/N1TSnxLU0DmnStVDrYbWGsXGf+kas0r2vLsd5ILEFk/T7s9gi+RZPJuXTRmE/MxhKJxrECXt3oXKlRSu+2KeKucIo8Jg44BcM9NWUQs4lvO5zJsJTqYreGsfKOJ2TRbYeI7WnqtW287iiRtbuxW0JY+QFcBVq4djSRaAK0tplSo5YmZylifB6sklziB53kzD0vuWJThzZVFpC4jXREsfL8Xceim+qIbjrQa/v4niaMz41rvPOmw26P0PFMLfEBrPBUWSAWx3hcmY5CKTVEmpylkHtCEYmGViQWxzO7BCyjG6CrrGC3hQFOSM7c5QUkGlt7FEuWhE1s/xHck8dhLGdYzHjdxOuOEts89hcGjAWSsDU5U2oU0+Qshbyzysm5/AxwuzABD67pRcS01pnKAsbnxr9sOq6yvK5jrvICSNhd8yQ7JeqPQTSBu1tleeNx4ZlRRmxXI3Y4hspuue9fgv+cmZkOQyk1RJqcpZCV68dVnNM1CddTU6rDmiorWAEv3rmVJ8w569wMO3FSvbNEcwd43bgnFJ1w3Du7Amwhtu3QyAeshsUYg7H09q7UaKX/e1MsfrCZ8Bu7AWePzfi2w0gkntmg1Gkv0dxB4ljohGPG58Eqzuka8uzkq6kk72+W9Vid6SrKwTU+n+iWg5xY4ktlE4nbhF7YSvzA0UyHopQaIk3OUizR1Er07X3YHVHcNWWQEOJbdcWmyqzIut2Enu5ZEzXnrxcSOLe66+vOpKuvshne+RPxTC+FhBY3zVYSixPbdgj7pGRcKTV6aHKWYu6KQgASB5vx1DgbG+u8M5VpThkNf4/jxn3ipPHI2t20P76+z54xz6Rx+M+c2uM8lUU6ayvqggClRi1NzlLMKs7F+NzEDzbjnlUCLqPbOKmMEhEnOcvtmZyJbdPx1CaitQcQEWK7mzCW6bd4qdhCbM9h7PbISIathqiz8LWu1lRq9NLkLMWMZXBVFBI/0AxeF+6ZxbqNk8ooicQhnsD01nNmWdgtIWL7jmAfbUfawrinjOvlKt2u1xEh9Mwmouv3j1TIahg0OVNq9NPkbAS4KwrBGCQSwz2vTGudqYyS1p41zrpzlReQONRCbJezW0D3Ehq9sXL9eKrLidYeIFrbexFblUEJ2ynn49EdApQarfR/7wjwVJc7ZQdwtnEK/2YzEophAp4MR6ZOR1a+n8C75+Aqzev1cff4AmKbDxJ9Zx+u8flYAe8pr+lfPhMJxQi/sgO8brwzylIdthoid2UR+R88J9NhKKWGQXvORkBnVXVwNkDHFmKbe9/DUKmRZnwePFNK+ky6XMl6Zybo1EIb0DUtQ2DlbFwVBYRf2Drk+WeJYx0kmlpP3VAppU4jmpyNkOiWg7Q9vBb3XGfFpi4KUJkSrz9G/KRCs91ZQS/uqSX4zpyKZ2rJgK9r3BbBv5pL8IIarBzfkGKLrt9Px1M9S3yooYvtaaLj2c1IXMudKDVaaXI2QozHjd0Swir2gsfSRQEqYyJv7SWydle/bYLnz8FbNX7Q1zYeN+5KZyeB2L4jPbaCOhWJJXTieooljrQT39kIVt8rbpVS2U3nnI0Q14Tk1jiNrbirx2nPmcoYuzV8wp6aI0Fsm8jrO7FDMfxnTMHkeDEBL67CIMbb921GYgnQieupFUuA2zpheoVSanTRnrMRYvm9WMU5xA8046kp00K0KiPEFqS99xpnqWQsi+DqeRivi/CrOwg9XUvHH992SsoAdkeE2O6e8y7dFYV4TlG6Qw2O9kYqNfrpW9YR5K4sIrqhDve8UkIPbcRuj2LlnHolnMoOEksgsQRWcPT+zqQ9AtJ3GY1UsvL85L5vCRKOIR1R7FAU17hcACJv7CG2/RBcUINnYnHXOb75E0c8rtONxBO6g4NSo5z2nI0gz7RSvLMrcM9xJlnHa3Xe2WjS/ujbtP3i1UyHMSx2q7O/YjqSM3BWcVpBL66SXDyTirsSW/+y6VhFOYSeqSXR1NbVXmzdQD3VjNvVa8FhpdToocnZCHKNy8W/fAae+eUAuihglLGPtDv/hmMZjmToXGX55Fy2CFfJyM45OxXjcTvDnj4PHU9uwG5zCuO2PvAy4dd2ZjS2sSawooqci+ZnOgyl1DBocjbCxBbIc0GOh7jOOxtVghfOA8A+2p625xQ7teUPjNuFqyQvK+YgWUEvwQtrkIQQfm2ns7l61Jm8rpRS6ji9K46wRP0xQk9swHvhFN3GaRQRW7CKgkD6kjOJxGj//VtEt9an7JqxnY29TsTPFFdhDjkXziOwogqSdbh0m6HUCj23hcgG3fdUqdFMk7MR5irPB68bz1llxHVYc9QIPb+F0AvbcE8eh/GP/IIAiSfoeHITdnMHJuAlurUeuyU07OtGNu4nuuVgCiJMHVdpHsbnQcJRQDfoTrX4gaPYx4b/t6OUyhxNzkaYsSxnYnS5n8T+FuyWcEqua3dEaf3688T3Nqfkeuo4ESFxsBnL5yb4V3PxTC8d2eezbUJP15JoaCGwchauklzCL29PSe+HtIbTthhgsGJ7jwBg/NnTc2aMudgYs8UYs90Yc2cvj3/PGPNW8mOrMaa522PXG2O2JT+uT2/kx2kpDaVGv7QlZ6e66XVr9z5jjBhjFqcrtpHmnjIOLIOrupD4puH3niUa2mg6/6e0fOYpWj715xREqLqzj3UgoRiuikIAJGE786NGgIgQen4r8bqj+M+ZiWdqKVbAi2fmeGLbG7CTvUtDunY0jkTiWZuceedOIOfSRbgnZ0edM2OMC7gPuASYC1xrjJnbvY2I3C4ii0RkEXAv8OvkucXAF4BlwFLgC8aYonTGn4zPGS7W5EypUS0tydlAbnrJdnnAJ4DRXb/gJO7KIrAM7oUlw16xGdvcSOPyHxNffwjf6umEfrWJ+I4jKYpUASQOOvtQuisKiO1qpPV/XkLaUtPjeTJjDK7iHHxnTcU7q6LruLemEhI2sdqhD0l2rogc6QK0Q2WMcYY4rdTdhozjo8aYp40x7ySPnWeMuWYApy8FtovIThGJAg8CV/TT/lrggeTnFwF/FpEjInIU+DNw8dC/kyGKJQAdKlZqtEtXz9lAb3pfBr4BjMwrYYYYt4vgexYQ+cPuYS0KiDy7m8az/wtpj1Hy7Icpuv9KcFu0ffflFEar4gebMTk+TK4fk+MDERJHOlL+PJ29Yr75k/AtmHTCY67CIO5JxURrDyDxRJ/XSDS39/m43ZpMzrK052yEfAm4EfgRMDl5bD/w6QGcWwns6/b1/uSxHowxU4BpwNODOdcYc7MxZq0xZm1jY+rnoHYuZBnNhZOVUulLzk554zLGnAlMEpE/9nehkb65jRR3aT6eWSVDXhTQ8fO3aVr9M1wVuZS+ehPeJZW4JuQT/OAC2v/7TRINbae+iBoQz5QSfAsmOT07RTlA6ldsJo600/bwOhL9XNc7rxKT48Nu731o024N0/HYekIvbuv1cffkceSuWYaV/B5OEzcAl4rIg0DnWPQuYHqKn2cN8CsR6Ttz7oWI/EhEFovI4tLS1M9ltPwecq88C8/0spRfWymVPlmxIMAYYwHfBe44VduRvrmNFBHBe+kUJHdwmxGLCC1ffpajH/wN3nMnU/rijbinHp/KkvupFRCJ0/6D11Id8mnLM6MM72xniNF4nGrr/SVRQxF5ey+I9NvD4RpfQM7lZ+AqCPR4TKJxOp7ciNg2voWTsdvCxHae2CtrjMEKeDGurPhvni4uoPOdSmdyltvtWH/qgO5dmBOTx3qzhuNDmoM9Vyml+pWuu/apblx5wDzgL8aY3cBy4JGxtCjAGIM1IQf3mSXYzQNb5i7ROM0f+R2tn3+GwIcWUvL4B7CKTnyh9swqwX/FbNp+8Bp2W2QkQj+tJJrbu4YDO7mKclLac5Y42k58dxPeORMwPk+f7YwxGGOQSJzEsePDqmILoWe3YB/rIHj+HFyFQSJv7SX03FYSja1d7SIbsq+MRho8BnzXGOMDZw4aznSJ3w/g3NeBKmPMNGOMFycBe+TkRsaY2UAR0H0+wRPAhcaYouRCgAuTx9Iq3thC+x/eInEkfYWTlVKpN+DkzBhzvjFmWvLzCmPMT40xPzHGlA/g9H5veiJyTERKRGSqiEwFXgEuF5G1g/puspxVnItrQg6RNw+csq3dHOLwJf9Lx/1vkffFVRTdfyXG23vJgdx/WoEcDdPx32+mOuTTTmTdHtqfWH/CMc+MMjzVA/kzH+BzvL0X3C5n0v8piAjtj71N+IXjQ5eRN/cQ338E//IZuCc4vaj+JdMxQS8dz25GYnEAYlvridcdTVnco8TtQAVwDCjA6TGbwgDmnIlIHLgNJ6mqBR4SkY3GmC8ZYy7v1nQN8KB0W8IrIkdwksDXkx9fSh5LK+mIOgm67lmq1Kg2mJ6zHwKd8yu+A3gAG2fibb8GcdMb07zznRfj+I7+FwXE9zTTuOK/iTy/h6KfXkn+F1bhdAD0znf2JLznTqbtOy8jsUFNgVHdiC3E64/hLi844bhnagm+eRNT8hx2a5j4ria8cyqw/H33mnUyxuCpLifR0EK8ocWJZ2aZs7pz9oTj7XxuAitnIW1hwi/vQESw27K3xtlISPaSlQBX4ywGWA7MEJH3ikhrvycnicijIlItIjNE5CvJY58Xke5vJr8oIj3KAYnIf4vIzOTHT1LyTQ1S1/9/Xa2p1Kg2mOqPlSKy1xjjxlk2PgWIAqfuBsK56QGPnnTs8320XTWIuEYNT3UJ7Q++jlXS9wTt6No6Dl/2ABKKUfLEB/GeNwVwelD6S9By/2kFRy5/gNAvNxL82wUpj/10YB9ph2gcd7K+WScRQTqiYMAK+ob1HFaen+B7FmAVBAd8jreqnMibewm/ssPZxLwgiGtBz/Pd4wvwLZxM5K29WONyISGnVXImImKMWQ/kiUgDcPrtl6alNJQaEwbTc9ZijBkPrAQ2iUjnBNtTv/1XgLNbgL2vA7sx5GyIfpLQI5tpWnk/xuei9KUbsabn0/7bN4gfbKbj0Xe6ek564//rKtxzSmj75osjVjB1rIvXO8XeXRUn9pwh0Pbw60Q39D+/W0QIr91FdHPv87w6fy/u8QUD6jXrZDwuvLMrsA+3EXl9V79tvQsn4ztzCla+MzfRyuu5mGCMexOoznQQmSKanCk1JgwmObsXZy7F/+IUlAVYAWxOdVBjmWV7CP/HBox1Yi9Y272vcuTKB3HXlFL6yk2YQg+hv9SC141VEMRuCRFZt7vPxMtYFrmfWkHs7UNE/rQjHd/KmJM4eAwrP9Cjd8xYBqsg2O+KzXhjC/GdjSQOtxF+eXuPlZMA4Ze2E35t55Bi89ZU4pldgXdORb/tjGXwLZwMtg0ug8nSArQj6C/A48aYLxpjbjTGfKTzI9OBpYMJeHGV5oH7tFqhq9SYM+D/wSLyDeACYEWyhhA4Ky5vGonAxir3vDLsQ+2EXt1JZMN+7EiM5tsf59jHH8N/xWxK/nIDYgkdT23CKgiQc2ENVtCLb+FkEvXHSPQzwTt43XysCXm0ffPFNH5HY4d/RRWB82b1+pirOAf7aN+FaGO1Bwm9soPgu+fgGp9P6LmtxPYdnw9ut4SIbasfcmyW30Pg7JkD7gnzTCkh74MrsPJPu+RsBU5ds5XAB4APJj8+kMmg0sVbNZ6cSxf1OwVCKZX9BrXjsIhs7fzcGHM+YIvIsymPagzz1Di12RJ7jhBrDxF+fjvRd/aTc/tyCr51IfaxEB1/3ogV8BK8cH5XqQXPrHIim+oIr9tNTmVRrzdf43WTe/tyWj71Z6Jr6/AuPvVqQHWcFfRCH3XHrKIcZHsDdjjWY0hSbJvYviN4JhVjPG6CF9TQ/vh6Qs/UYi6swV1eSOTtfWBZeFO0sGAgTscXaBE5P9MxKKXUcA2mlMazxpgVyc8/jbMF0/8ZY+4aqeDGIneNU7nb3thK5Le7Sextxf+BWbiWF5OoP4aV48VdUUDw4vknFCg1Lgv/GVOwjzg1svqSc/NZmHwfbd96acS/l7EktqeJyKa6PoeN+9spIFHf4iwkmOJs4G28boIXzsMqCCDhGHZriNiOQ3hnleu2OmmQrDX2IWPMZ5L/pn0D8kwJvbydjqc2ZToMpdQwDWZiwjyc+mMAHwXOx1mqfkuqgxrLXBPzMfk+Wr/0LLEn9+JfPtN5IQ96MX4Pxuch+O65vW5W7Z5ein9FFe5JxX1e38r3k3PrYt0QfZCiW+qJbanvs7fJVZJL4LxZWIU9V0nG9x4Gl+VscJ9k+T3kXHYGnqmlRN7ZD5bBOz99vWanK2PM2cAOnPvSAuDvgB3J42Oe3RrG7uh9uy+l1OgxmOTMAsQYMwMwIrJJRPbhVMpWA2SMwbt8Itb4HEqevYHgFXNwVxaRc+kiXONyT31udTnG3f9KrNyPLxv2hugi0lXMdKwT2yZx6Biuk+qbdWd8HjwzyrACPXu+Es3tuCuLevxeOhd9+OZPJLCiethlONSA3A38vYicIyLXisgK4Fbg+xmOKz1iCV2pqdQYMJg5Zy8AP8Cpvv0bgGSi1vcYm+pV8UNXg8tg5Q7txTped5TIm3sIXjS/1xtx9w3R876wEldZ/0lfb6Lr9xN5Zx+5V5415DgzJbJ+H3Z7BLsljP/MKbhK8vptn2hsg7jdo75Zj3bNHdjHOvBMKTnhePCi+RC3+zzPyg90lbZQI64aeOikY78C/j0DsaSdxOJYfv1bU2q0G0zP2Q1AM/AO8MXksdnAPakNaeyzCvzDS3g8LhKNrUQ39l13K/f/nTPkDdHFFqK1ByCWILy2/7pa2Si27RD24Xbs5g5Cz27pd9cEESGW3H+yv54zgNiWg4Se29JjXpoxRnsrssc2nO2VursaZ6hzzBPtOVNqTBhMKY3DInKXiHyhswCtiPxRRO4eufBUb9xl+bgnj3NKcYRjvbbxzC4d+obotuCdNxF3ZRHxXY3E648N6LTYrkZiew9ntAiuHYpiHwvhnlxM4Lxq7JYQ4Vf7eV0WQWIJvAsmnbIwrFWUA3Eb6bYxevtj7xB5Z1+qwlfD94/AD4wxrxhjfmGMeRVn67mPZziutHCPL8AqGXxPuVIquwxmtabHGPMvxpidxphw8t9/SW5krtLMd9ZUiCcIP7cFsXsfUhvqhujGbeGrqSTw7jlYRUGkY2DJXbT2AKGnNtHxx7eJHziasiRNRAi9vL3fIrCdEoecXRRc4wtwlxfiXTCJ2LZDxHY39rimROIYyyJw/hx8Z0455bU7V2x2xmG3hEjUHwOXFvzMFiLyEjADZwrGOpzi2TOTx8e8wHmz8M3VEjpKjXaDeVX5Jk4R2luAhcl/3w18YwTiUqfgKgziP3sm8bqjfZbWGMqG6HZbmOjWeiSewLhd5FxxJp7pZX22Txxpp/1PG7A7ogQvno9/RRV2R5SOJzbQ8fh6Ek0D2m/6lBIHmoluOvU2rolDTrLUubjCd8ZkrJI8Im/v60oWRYTwS9tpf/RtZxjIMgOqCda5UrOznEZs72EAPJPHDel7UqlnjKkEEJGfi8g3ReTnOAuZJpziVKWUyhqDSc6uBi4XkT+JyBYR+RPwXuCakQlNnYp3VgXB9yzAPa20zza5/7SCxN5jhH65cUDXjG6pJ/zSNiTirNQ0xjjzsnY0IJETh1AlliD0l1rsI21gnC2kvNXl5L5vMf5lM7CPdXRdZyhiOxqIvLMPYwxWYZB43QB644zBPbEIk+zNMpZF8Pw55Fy8oOt7Cb+0ndjWeqcu2SC2uTEeFybPTyK5U0B872Gs4pzTanPxUeC3wMk1SyaSXMQ0ltnh/9/enYdHWZ2NH//es2VPWBP2fUdBlstHNnUAACAASURBVIILCtWqQCtWa1XEiktFrdby+utrXV6X1mqtfVu1ilVRXFoUi/pWrChF64IKCgiigOw7SELCkj2z3L8/ZpIOSyAhs2Xm/lzXXMycZ7uPJif3nOc553gpfXkRNet2xzsUY0wTNSY5q69rIfWmIU8groI8RAT/3nK8Gw5fz7ExC6KrPxBMWjq1wpH1nwELgQOVVC5YQ9WyLf/ZV5WqhesJHKgkY3S/g6aYEKcDz4AOZF/0HZwdjj4C8ohxqFK9bAuVH60JJmSBAK7OrdDy6qMuoQSQPqIHmWcOOKjMkZ2GpLlQn5+yV5fgXfstnsGdSRvStdGz6GeeNYD0U3oRqKzBv/uA9Zolnj6q+lV4QehzvzjFEzteP1rlBeL3zKcxJjIak5zNBt4UkXNFpL+IjCX4LXV2dEIzjVG9fCuVC9bg3X7wxLONWRDdt7UYrfLi7nfw4trOvEzcfdvj/WYX/pLQLb11u/FuKCTtpC71TkFRO+9X5SfrqA5L7I5GfQGqPlpD9fKtuHvlk3nOCYjDgatTcOJd37bi+o89RvJZvWwrWlaF56Qux5WYQfC5M0e6G1TxDOyIq1ubYx9kYqlIRHqFF4Q+1/+DkyRqH12w0ZrGNH+NSc5uBd4FpvGfB23fB/47CnGZRsoY1RtHyywq31992HNemZc1bEH0mjW7kOw0XB0On1c4bUhXcLuo+mxDcKqNlTtwts/DM6jLUc8pImi1l+pVO4/53JsGlIr5X+PdWETasG6kj+pTd3vSkenB0Sb7qCNHq7/YQtn/LUUDR07S0oZ0JXP8YNJO6nLc604GKqqpXr4FfAHSR/SoGyRgEsYM4DUR+YGIDBCR84DXgGfiHFfU1U4aLe5GLZlsjElAR/0tFpEzDyn6IPQS/tN3Pgr4d6QDM40jbheZZ59A+VvLqZi/ksxxg3CGHmCXtGMviK4+f3BKiT7t6ma2D+dId5M+tCtVizbg21pM1vjBaCBwxH0P5TmhE74txXjXfYvnaCPJBDx920Gfdrh7Hj4IIfO7A5CjrE3p370/+FxYPTGJy4GrIPeY8R6Nev1UL9tKoKya9FN7N6j+JqYeBLzA/wKdga0EE7OH4xlUTNR++TnGCiLGmMR3rK9Yz9ZTXpuY1SZpPSIWkTlujkwPmWefQMXbK6hZsY2MM/rWbcuaMozS+z6i7A+f0uqVHx92rLicZJ83pN5eJwB33/b4du4LJkBprgY/bOjKz8XZNofqlTtx9+tQf/IkctSRoUebuFd9AfxFpXgGRHdQniMnOPu6d91u3L3ycbVr/DN1JqpGA6+q6h9EpD3B0eQnAPnAt3GNLMok3Y2rR1scR/kCY4xpHo6anKlq91gFYiLD2SKTrPNOQtKCE6rWzhheuyB62R8+xbehBFfP/yyerv4A+ALBhOsoPUHiEDLPGlDv9qPxnNCJyvdX49u6B3e3w0eXVi3agOSkkzbw6HM0VX+5Fa3xkf6dg78P+PeUQkBxFhx9lv+mCv/v48yP7rXMcXkCODf0/o+hf73A08CEuEQUI842OWSOTv5xD8akAps9Mwk5stMRtxP1+il/68u658TqWxDdu3kPpa98hn/f0UdCNoWrS2s8gzvjbH34Ope+ogPUrN6JVtQc8zyB0tA8bIdMvOvfHXwWram3LRsi85wTyPhuf7ulmZg6qupWEXEBY4EpBBc+PzW+YRljTMNZcpbMnA5c7fOoWRWcud/RNrNuQXR/YVndbt41u3BkpeHIi96CyeIQ0od2O2xOMFUN9ppleEg7qfMxz+Pq3Apq/PgLDxxU7myTg2dQ57oew2hydWyJ20ZpJqoDIlJA8Pbmytql5oDo/2DEWfWKbRz466f1rhhijGk+LDlLYuIQ0kf2JP2Unvh2lFD+5nLSftwLqoILold/uZXKj9cG5+vq0+64RzA2hm/3gYPWovSu201gTxnp3+neoFFmrg4twCH4th08ZYirY0vSh3WLdLim+XkMWAzMJDiyHOA04Ju4RRQjWuODQABxWLNuTHNnv8UpwNOvA5lnnxBcA9IlpJ/fl7LHP8e3Yx++bSU48jJw9y6ISSy+7SVUL91M4EAl6vNTvXQzzoJcXD3qX+UgnLhdONvlHZScBSpr8O+viOuC6yYxqOrvCS4zd5qqzgoV7wB+Gr+oYqP2+VJjTPNnE+KkCFfHlmR3DM5f5szMoeqNNej6CnJuPjmmcXj6d6Dm6+1Ur9xBxim9yDhzAOJxNqrXzt29Lb5d+1B/AHE68G4sovrzjWRfPALJqn9Ep0kNqrr2aJ+PJjS59qOAE3hGVR88wj4XA/cSHKn+papeFir3A7WrE2xV1dgOQPD6wZIzY5KCJWcpKHxB9KwbvhPTb9uOTA/unvl4135L2pCux/UAv6dPOzx92tV99n+7H8lJP2jJKWMaS0ScBG+Fng1sBxaLyBxVXRW2T2/gdoI9c3tFJHzul0pVPSmmQYexnjNjkofd1kxRjV0QPZI8AztCQKn896pj71wPVSVQXo2q4t+9PyajNE3SGwGsV9WNqloDzALOP2Sfa4FpqroXQFUPX9A2TlwdWxx1nkBjTPNhyVmKasyC6JHmbJlF2vBueAYde3RmfWq+3EbZq4sJ7ClDq31Rn9/MpISOwLawz9tDZeH6AH1E5BMRWRS6DVorXUSWhMp/eKQLiMiU0D5LioqKIhq8p18H0prwO2WMSRyWnKWoxiyIHg1pJ3bG3anVsXesh7MgFwJat6C6JWcmRlxAb2AMMBGYLiK1y0R0VdXhwGXAIyLS89CDVfVpVR2uqsPbtm3YIJiGUl/ABsUYkyQsOUthtQuil/5uAYGy6niH0yjOgtzgw89OBxln9seRm37sg4w5uh0E1+Os1SlUFm47MEdVvaq6CVhLMFlDVXeE/t1IcA3iIdEOOFzZa4up+mRdLC9pjImSmCVnIjJWRNaIyHoRue0I268Xka9EZLmIfCwix7dOkGkwSXOR88tTqflwC7vyHqTwpL+w9/o3KX9+Gd41exJ6MktxOHB1bIm/qBRXl9YxmaPNJL3FQG8R6S4iHuBSYM4h+/yDYK8ZItKG4G3OjSLSUkTSwspPA47/ocrjoF4/YoueG5MUYjJasyGjoICXVPXJ0P4TgD8RXH7FRFHW1JNxDWxLzSfbqFm4jcqXv6biqaUASMt0PCM74Tm5E55TOuEZ0RFHi+itItBYrnZ5+Dbvwb+nFFdbGxBgmkZVfSJyEzCP4FQaM1R1pYj8BliiqnNC284RkVWAH/hvVS0WkVOBp0QkQPBL74OHtG/Rjh18NpWGMckiVlNp1I2CAhCR2lFQdY2Xqoavx5NFcA4hE2UiQvo5vUg/pxcAGgjg+2YPNYu2U7NwOzWLtlP66w/q/m+4+rcJJWud8ZzcCdeAtogzPnfH3b3y8e+rwJFttzRNZKjqXGDuIWV3h71X4JbQK3yfT4ETYxHjEfkCoNhUGsYkiVglZ0caBTXy0J1E5EaCjZ4HODM2oZlw4nDgHpCPe0A+WVcPBSBwoIqaxTupWbiNmkXbqZqzhornlgf3z/bgGdERzymdyLjsRNwDYjeUX9wuMk7pFbPrGZOo1OsHLDkzJlkk1CS0qjoNmCYilwH/A0w+dB8RmQJMAejSpUtsA0xRjtx00s/qQfpZPYDgLRT/hpK6nrWaRdspffBjyp9eSsHmqTgyPXGO2JjUIk7BM6gzzjY58Q7FGBMBsUrOGjIKKtws4C9H2qCqTwNPAwwfPtxufcaBiODq1RpXr9Zk/mQwANULtrDnjOeoeHYZ2T8/rFPUGBNFkuYmfVi3eIdhjImQWD0sdMxRUKFlUWp9H7Ax4c1I2uld8ZzeJTipbY0v3uEYk1LUFyBQVYMG7PuqMckgJsmZqvqA2lFQq4G/146CCo3MBLhJRFaKyHKCz50ddkvTJLacO07Hv/0AFX9bEe9QjEkpvl17KXv5MwLFZfEOxRgTATF75qwBo6B+EatYTHSkndsL99D2lD74MZmTT4rbKE5jUk5oQIBNpWFMcrC/niZiRCTYe7auhMpXYzr/pjEpzUZrGpNcLDkzEZV+QT9c/dpQ+sACW+fPmBix5MyY5GLJmYkocTjIuX0UvhW7qZ5rYzqMiQW125rGJBVLzkzEZUw8EWfXPErv/8h6z4yJAVf7FqQN62ZrzBqTJCw5MxEnbifZt54WnKT2w83xDseYpOdql0faoM7H3tEY0yxYcmaiIuvqITgKsih9YEG8QzEm6QXKqwmUV8c7DGNMhFhyZqJC0t1k/79TqZ6/kZrFR1sMwhjTVFWfrqPiPRshbUyysOTMRE3W9cORFunWe2ZMlKnXbyM1jUkilpyZqHHkpJF980iq/vEN3pWF8Q7HmKRlyZkxycWSMxNVWTePRLLclP7Oes+MiRb1+m0aDWOSiCVnJqqcrTPJun44lS9/jW9jSbzDMSY5ef2Iy5IzY5KFJWcm6rJvOQVcDkof+iTeoRiTlNKGd8fdMz/eYRhjIsSSMxN1zg65ZF09hIrnluPfcSDe4RiTdDy9C3C1y4t3GMaYCLHkzMRE9n+fCv4AZX9aGO9QjEkqGgjgLyolUOWNdyjGmAix5MzEhKtHKzImnkj5k0vwF1fEOxxjkoZWeCn/53J8W4vjHYoxJkIsOTMxk3P7qOAfkkcXxTsUY5KG+nwANiDAmCRiyZmJGfeAfNIv6EfZY58TOFAV73CMSQrq9Qff2FQaxiQNS85MTOXccTq6r4ryJ5fEOxRjkkMoObNJaI1JHpacmZjyDO9I2jk9KfvTQrTSHmA2iUVExorIGhFZLyK31bPPxSKySkRWishLYeWTRWRd6DU5VjGrJWfGJB1LzkzM5dxxOoHd5ZTPWBaV86s/gG+DTXhrGkdEnMA0YBwwAJgoIgMO2ac3cDtwmqoOBKaGylsB9wAjgRHAPSLSMhZxO9vkkDG6L46c9FhczhgTA5acmZjznNEVz6mdKXvok/88LxMhvvXF7BnzPLt7/ZnqBVsiem6T9EYA61V1o6rWALOA8w/Z51pgmqruBVDV2kVjzwXmq2pJaNt8YGwsgnZkpeHukY94XLG4nDEmBiw5MzEnIuTceTr+rfupmLkiIufUQICyxz6jcNBf8H61G8lyUz59aUTObVJGR2Bb2OftobJwfYA+IvKJiCwSkbGNOBYRmSIiS0RkSVFRUUSCDhyoxLdrH6oakfMZY+LPkjMTF2njeuMeXEDZgx+j/kCTzuXbWMKeM19g/81v4xnTjYKVN5Jx+SCqXl1FYL+NCjUR5QJ6A2OAicB0EWnR0INV9WlVHa6qw9u2bRuRgGrW7aZi3lcROZcxJjFYcmbiQkTIvuN0fGuKqfq/1cd1DlWl/MnFwd6yL3bR4tkJtH5rEs6OuWRdMxSt9FH5sv3RMg22A+gc9rlTqCzcdmCOqnpVdROwlmCy1pBjo8PrB7cTEYnJ5Ywx0WfJmYmbjB8NwNWnNaUPLGj0LRnfln0Un/NX9t3wFp5TO5P/9c/Iunpo3R8o9/AOuE7Mp/zZ6Aw6MElpMdBbRLqLiAe4FJhzyD7/INhrhoi0IXibcyMwDzhHRFqGBgKcEyqLOvX6bQJaY5KMJWcmbsTpIPu2UXiXfUv1O+sbdIyqUv7MUgpPfIKaRdtp8dQPaD3vJ7i6HHxnSUTIumYo3iU78a74NhrhmySjqj7gJoJJ1Wrg76q6UkR+IyITQrvNA4pFZBXwPvDfqlqsqiXAfQQTvMXAb0Jl0Y/b67NpNIxJMpacmbjKnHQizs65lD6w4Jj7+rfvp3jc39h37Zu4h3cg/6sbyJoyvN7bORmXDwKP03rPTIOp6lxV7aOqPVX1/lDZ3ao6J/ReVfUWVR2gqieq6qywY2eoaq/Q67mYxez1g9tGahqTTCw5M3ElHhfZt55Gzcdbqf5o8xH3UVXKX1jO7hOeoGbBVvIeH0+bd6/A1e3o00g5W2eS8cN+VPxtBVrti0L0xsRf+rBupI/oEe8wjDERZMmZibusa4biyM86Yu+Zf+cBSia8zL4r/4F7UAH5K24g+8YRiKNhP7qZVw9BSyqpfOObSIdtTEJwtsnBVZAb7zCMMRFkyZmJO8lwk/1fJ1M9bwM1S3cCwd6yipkr2H3CE1S9u5G8h8+lzQdX4urZqlHnTvteD5ydc6mwW5smSXm3FuMvKYt3GMaYCIpZcnasNetE5JbQenUrROQ9Eekaq9hM/GXd8B0kL43SBxbg311GyYWvsPfy13H3a0P+l9eTPfWUBveWhROng8yrhlA9fwO+LfuiELkx8VX18Vpq1tigF2OSSUySs4asWQcsA4ar6iDgVeChWMRmEoMjL53sn4+k6vXV7B4wjaq315H7h7Nps+Bq3H3aNOncmVedBEDF88sjEaoxCUW9fhutaUySiVXP2THXrFPV91W1IvRxEcFJHE0KyfrFSKRlOq6eLclfdj05vzwNcTb9R9TVrSVpZ/Wg4rllaKBpqxEYk0jUH4CAWnJmTJKJVXLWoHXnwlwDvH2kDdFYm84kBmebLNpt+S/aLvop7v6RWdqmVuY1Q/Bv2U/1vzdF9LzGxJN6/QCWnBmTZBJuQICIXA4MB/5wpO3RWJvOJA5HTtpxPVt2LBk/7Ie0TLeBASa5hJIzm+fMmOQSq+SsQevOicj3gDuBCapaHaPYTAqQdDeZkwZR+X+rCZRUHPsAY5oByfCQ+f3BuDodfc4/Y0zzEqvk7Jhr1onIEOApgolZYYziMikk85ohUO2nYqYthm6Sg7gcuPJzcWR44h2KMSaCYpKcNXDNuj8A2cBsEVkuIocuOGxMk3hOao97aHsqnv2i0QutG5OIAmVV1Kz7lkCVN96hGGMiKGYPKqjqXGDuIWV3h73/XqxiMakr85oh7L9xLt5lu/AM7RDvcIxpEn9RKVUfryPrhzmQ7o53OMaYCEm4AQHGRFPmZSdCussGBpikYKM1jUlOlpyZlOJokUHGj/pTMXMFWmm3gkzzZsmZMcnJkjOTcrKuHoLur6by9dXxDsWYpqmbSsOSM2OSiSVnJuV4xnTD2b0F5c9+Ee9QjGkS9frBKVGZG9AYEz/2G21SjjgcZF49hJr3N+PbUBLvcIw5bp4TO5J13pB4h2GMiTBLzkxKyrryJHCILYZumjVHugdny6x4h2GMiTBLzkxKcnbKI+3cnlQ8vzy4eLQxzZB38x68G22NYWOSjSVnJmVlXTMU//YDVP9rQ7xDMea41KzaQc03O+MdhjEmwiw5Mykr/bw+ONpk2sAA02wFSqtw5KTHOwxjTIRZcmZSlnhcZPxkEFVz1uAvKo93OMY0ivr8aEUNjpyMeIdijIkwS85MSsu6Zih4A1T89ct4h2JMowRKqwBw5FrPmTHJxpIzk9LcA/Nxj+xIxbPLbDF006zUJWfWc2ZM0rHkzKS8rGuG4ltVhPfzHfEOxcSZiIwVkTUisl5EbjvC9itFpEhElodePw3b5g8rnxPtWF2dW5E9cSSOVjaVhjHJxpIzk/IyLhmIZLptYECKExEnMA0YBwwAJorIgCPs+oqqnhR6PRNWXhlWPiEG8eJI9yBOa8aNSTb2W21SniM3nYyLB1I562sC5TXxDsfEzwhgvapuVNUaYBZwfpxjqlf1V9upWfttvMMwxkSBJWfGAJnXDEFLa6icvTKi56184xu+7fowB+59P6LnNVHREdgW9nl7qOxQPxKRFSLyqoh0DitPF5ElIrJIRH54pAuIyJTQPkuKipo2eax3zS58O/Y26RzGmMRkyZkxgOe0Lrj6tKbi2WUROZ+/qJySS2dT8sNZ+AvLKXvoE/y7yyJybhNXbwLdVHUQMB94IWxbV1UdDlwGPCIiPQ89WFWfVtXhqjq8bdu2xx2EBpRAWTWOXBsMYEwysuTMGILP72RePYSaj7fiXbvnuM+jqlTM+orCAdOofH01Ofd9l/ylU9BqP2X/+2kEIzZRsAMI7wnrFCqro6rFqlod+vgMMCxs247QvxuBD4CorUiu5dWgahPQGpOkLDkzJiRz8mBwChUzjq/3zL+rlJILZrF34ms4u7cg/4vryP2f0bgH5JNx2YmUP7EYf6H1niWwxUBvEekuIh7gUuCgUZci0j7s4wRgdai8pYikhd63AU4DVkUr0MCBSgBLzoxJUpacGRPibJdD+vf7UPHCl6jP3+DjVJXy55exe8A0quZtIPcPZ9P202twn1BQt0/O/5yBVvko++PCaIQeU1XvrKNw2FMUjXmO6g82xTuciFFVH3ATMI9g0vV3VV0pIr8RkdrRlzeLyEoR+RK4GbgyVN4fWBIqfx94UFWjlpxptRecDpvjzJgkJc154s3hw4frkiVL4h2GSSKVc76h5PxZtHrjUjIm9Dvm/r6t+9g35U2q523AM6oLLZ6dgLtPmyPuWzLpNare+IaCzVNxtml+c1P51hez/5Z5VL25FmfPlmilj8DOUtLO7E7Ofd8l7dQuMYlDRJaGnu1q1praftW23SISqZCMMVHW0PbLes6MCZM+vjeOdtnHHBiggQDlTy6mcOAT1Hy8lbzHxtHmwyvrTcwg1HtW4aXsT82r9yxQVs3+299l98AnqH5/M7m//x4FK2+k3fqbyXv4XLxfF7LntBnsGf83apbujHe4KUNELDEzJklZcmZMGHE5yZw8mKq31uL/tvSI+/g2lLDnrBfZd8NbeE7uRP7XPyP7ppGI4+i/Tu7+bcm4eCDlj32Ov7giGuFHlKpSMXMFu/s+TtmDH5Nx6QkUrP05ObeOQtJcSIab7KmnULDxF+Q++D1qFm2naPjTFF84C+9Xu+MdflKr/HgtNatsRQtjkpUlZ8YcIvOqIeBXKl48eDF09Qcoe3ghhSc+gfeLXbR4ZgKt//UTXN1aNvjcOXeNRstrKH84sXvPar7YyZ5RM9h7+es422fT5tNraPXCBTjb5xy2ryPLQ86vRtFu01Ry7h1D9XubKBz8F0omvop3zfGPfDVHpqp4NxXVra1pjEk+lpwZcwh33zZ4RnU5aDF07+oiikbNYP8t80g7qwcFq24k65qhjb6t5B6YT8ZFAyj782cEShKv98xfVM7eKXMoGv40vnXFtHh2Am0/v5a0Uzof81hHXjq594yh3aZfkH3bKKreXEPhgGnsvfL/8G0siUH0qUGrvOAL2EhNY5KYJWfGHEHmNUPwrS2m5sPNlP5uAYUnPYlvbTEt/3YhreZMxNkx97jPnXPXaLS0hrJHFkUw4qZRr5+yRxexu/efqXhuOVlTT6Zg7c/JunroMW/XHsrRKpO8B75HwcZfkD31ZCpeWcnuvo+z9/o38W3bH6UapI7aaTTERmoak7QsOTPmCDJ+PBDJ8bBn7N84cMd7pE/oS8GqG8mcNKjJD2G7Tywg/cL+lD36GYF9lRGK+PhVvbeRwpOeZP/Ud/CM6Ej+l9fT4k9jcbRo2h9/Z342eX88l3YbbibrumFUzFjG7l5/Zt8v3q73eT5zbLW3Mx251nNmTLKy5MyYI3BkeciaMgxHywxavXoxrWdfjLMgO2Lnz717NHqgmrJHP4vYORvLt3kvxT96heLvvYhWemn1j0tpPe8nuAfkR/Q6zg65tHj8+xSsu5nMKwZTPu1zdvd4lP23/gv/nvKIXitVOHIzcGRbcmZMsrJ5zoypR7TnkSq+YBbVH2ym3eapOPJi94c2UFFD2YMfU/qHTxGHkHPH6WT/v1OQdHdMru9bX8yBX39I5cwVSJaHli9eQMYF/Rt8vM1zZoxprhJunjMRGSsia0RkvYjcdoTtZ4jIFyLiE5GLYhWXMfWJ9jxSOXePRvdVUfbn2PWeVS/YQmG/xym97yMyLuhHwZqbyLnzjJglZgCuXq1p9dcLyf/6Z6R/vzfuwQXHPsgYY1JITJIzEXEC04BxwABgoogMOGS3rQSXQnkpFjEZE2+eIe1JP68PZQ8vJHAg+tMieNfuoXjCy5Duos1HV9HqpYtwdsqL+nXr4x6QT6tZP8bVo1XcYmiOyt5cTvVKm+PMmGTmitF1RgDrVXUjgIjMAs4nbGFgVd0c2haIUUzGxF3OPWOoGv405Y9/Ts4dZ0TtOoG9lZSc9zLictCmkXOzmcShNT4Ce0qha+t4h5I0Dhw4QGFhIV6vN96hmGbO7XaTn59Pbu7xj+avFavkrCOwLezzdmDk8ZxIRKYAUwC6dInNWn7GRItnWAfSvt+b0j8uJOvnI3HkpEX8GurzU3LJbHyb9tLmvcmWmDVjNlIzsg4cOMDu3bvp2LEjGRkZthyWOW6qSmVlJTt2BHu1m5qgNbvRmqr6tKoOV9Xhbdu2jXc4xjRZ7j1j0JJKyqd9HpXz779lHtXzN9LiyR+QdnrXqFzDxEagNDj1isPmOIuIwsJCOnbsSGZmpiVmpklEhMzMTDp27EhhYWGTzxer5GwHED7FeKdQmTEpz/OdjqSN60XZ/35KoKw6oucuf2oJ5Y99TvYtp5B19dCIntvEXu2zibY6QGR4vV4yMizRNZGTkZERkVvksUrOFgO9RaS7iHiAS4E5Mbq2MQkv9+7RBIorKX9iccTOWf3BJvbdNJe0cb3IfejsiJ3XxI9kenB1aY14YvVESvKzHjMTSZH6eYpJcqaqPuAmYB6wGvi7qq4Ukd+IyAQAEfmOiGwHfgw8JSIrYxGbMYnAc3Jn0s7pGew9K69p8vl8G0oo/tHfcfVuRauXL0Kcze4JBnMEnl4FZJ516EB3Y0yyiVmLrapzVbWPqvZU1ftDZXer6pzQ+8Wq2klVs1S1taoOjFVsxiSCnHtGEyiqoPzJpk1MGjhQRfF5wRlpWs+ZGNMJbk10NedJw40xDWdfp41JEGmndiHtez0oe+gTAhXH13um/gAlE1/Dt66E1q9ejKuXTbmQLNQfoPRvn1Kzyh7XNSbZWXJmTALJuWc0gcJyKp5aelzHH/jVfKrnrqPF4+NJ+273CEdn4ilQVgW+ANjzZga48sor61YxcbvdtGnTBsIkuAAAErtJREFUhlGjRvHQQw9RXm5r1jZ3lpwZk0DSRnXF891ulD70CVrZuBE/5c8to+yPC8m6aQRZ1zX7pSfNIf4zUtNGF5qg008/nV27drFlyxbef/99Jk2axOOPP87QoUPZvXt3vMOLm5qapj+3G2+WnBmTYHLvGUPg2zLKpze896z64y3su+5N0r7Xg7yHz41idCZetHaOM5uA1oR4PB7atWtHhw4dOPHEE7nhhhtYuHAhRUVF3HbbwUtYP/bYY/Tr14/09HR69+7N/fffj8/nA+DOO++kb9++h53/hhtuYNSoUfVef/78+YwZM4ZWrVqRl5fH6NGj+fzzg+drLCsrY+rUqXTu3Jm0tDS6devGAw88ULe9sLCQq666ioKCAtLT0+nbty8zZswA4IMPPkBE2L59+0HndLlcPP/88wBs3rwZEWHmzJmMHz+erKws7rrrLlSVa6+9lp49e5KRkUGPHj244447qK4+eLqid999l9NPP53MzMy6OmzYsIEPPvgAp9PJtm3bDtr/xRdfJC8vL+q9k9Y/bkyCSRvdDc/orpT+/hOypgw75qLkvs17KbnwFVzdW9Lq7z9GXM4YRWpiKXCgClyOmC5Sn6rK315xWJm7Wxs8/TugPj8V8w+fTMDdqwBP7wICVV4q31992HZP3/a4e7QlUFZN5YI1h23PGjcoIrF37NiRSZMm8eKLL/Lss8/icDi49957ee6553jkkUc46aSTWL16Nddffz1VVVXcd999TJ48mQceeIDPPvuMkSODi/dUV1fzyiuv8OCDD9Z7rbKyMn72s58xePBgfD4fDz/8MGPHjmXdunW0bt0aVeUHP/gBW7du5bHHHmPQoEFs376dNWuC9a+srGT06NFkZGQwc+ZMevTowfr16ykpKWl0vX/1q1/x+9//nmnTpgHBwTP5+fm89NJLFBQUsGLFCq677jrcbje//vWvgWBidu655/Lzn/+cxx9/nLS0ND755BO8Xi9jxoyhd+/ezJgxg3vuuafuOtOnT+eyyy4jKyur0TE2hiVnxiSg3HvGsOfMFyh/5guyb6p/pbNAaTXFE15GvQFavTkRR0u75ZWsnPk5eNxOm5fLHNPAgQM5cOAAe/bsITs7m4ceeojXX3+dsWPHAtC9e3d++9vfcvPNN3PffffRp08fRo4cyYsvvliXnL355ptUVlZy8cUX13udCy644KDPTz/9NK+99hrvvPMOkyZN4t///jcffvghixcvZvjw4KMWPXr04IwzgusIv/TSS2zatIn169fTqVOnuu3H47rrrmPSpEkHld1///1177t168aGDRt44okn6pKzX//614wbN45HHnmkbr9+/frVvZ8yZQqPPvood911Fw6Hg2+++YaPP/6YP//5z8cVY2NYcmZMAvKM6YZnVBdKH/yYrGuHIWmH/6pqIMDey1/Ht6qI1m9fjrtPmzhEamLF3SMf6zOLjaP1YonLedTtjnT30bdnp0Wsl6w+tVOuiAgrV66ksrKSH/3oRwcl9n6/n6qqKoqKimjbti2TJ0/mrrvu4pFHHsHtdvPiiy8yYcIEWrRoUe91Nm3axN13383ChQspLCwkEAhQUVHBli1bAFi6dCktW7asS8wOtXTpUgYMGFCXmDXFiBEjDiubPn06zzzzDJs3b6a8vByfz0cgEDjo+kfrGZw8eTJ33nkn8+bNY9y4cTzzzDMMGzaMIUOGNDneY7FnzoxJQCISHLm5o5TyGcuOuM+B//k3VXPWkPfwWNLP7hnjCJOTiIwVkTUisl5EbjvC9itFpEhElodePw3bNllE1oVekyMZl6oSqKyxec5Mg6xcuZK8vDxat25dl4zMnj2b5cuX172++uor1q1bR6tWrQC49NJLKS0t5a233qKoqIh33nmHyZOP/mNce8ty2rRpLFq0iOXLl5Ofnx+xB/IdjmCKEv5z7/f7D0qwah16m3H27NnceOONXHLJJcydO5dly5Zx9913N2pppdatW3PRRRcxffp0ampqePHFF5kyZcpx1qZxrOfMmASVdlYPPKd2pux3C8i6eshBvWcVf/uSst99TOZ1w8i66fBvjKbxRMQJTAPOBrYDi0VkjqquOmTXV1T1pkOObQXcAwwHFFgaOnZvJGLT8hrKZn9O+qm98PRtH4lTmiS1Y8cOZs6cyYUXXojD4WDgwIGkp6ezceNGxo8fX+9xLVu25LzzzuOvf/0rW7dupVWrVpx7bv2Di4qLi1m1ahVz586t22/79u0HLfo9bNgw9u7dy5IlS47YezZs2DBmzJjB9u3bj9h7lp+fD8DOnTvp3Dm4PPfy5csb9CXlo48+YsiQIdxyyy11ZZs3bz7s+v/617+4+eab6z3Pddddx3e/+12eeuopKisrmThx4jGvHQnWc2ZMgqrtPfNvO0DF88vrymsWbWPvT+fgGdONFo+Nt2eQImcEsF5VN6pqDTALOL+Bx54LzFfVklBCNh8YG6nAArUjNW3BcxOmpqaGb7/9lp07d/LVV1/xl7/8hVNOOYX8/Hx+97vfAZCdnc0dd9zBHXfcwbRp01izZg0rV65k1qxZ/OpXvzrofFdccQX//Oc/efLJJ5k0aRJOZ/2Di1q2bEnbtm2ZPn06a9euZeHChUycOPGgheTPPPNMTj/9dC655BLeeOMNNm3axCeffMIzzzwDwMSJE+natSsTJkzg3XffZdOmTbz33nu88sorAPTq1YuuXbty77331j3v9V//9V8NavP69u3LV199xRtvvMGGDRt49NFHef311w/a56677uLtt99m6tSprFixgjVr1vD888/XDVgAGDVqFH379uWXv/wll156KTk5Oce8diRYcmZMAks7uyfukztR+sACtMaHb9t+in84C2fHXFq9ejHitpGZEdQRCB83vz1UdqgficgKEXlVRDo35lgRmSIiS0RkSVFRUYMDC5TaHGfmcAsWLKB9+/Z06dKFMWPGMHPmTG666Sa++OILCgoK6va76667+NOf/sT06dMZPHgwo0aN4uGHH6Zbt24HnW/cuHHk5eWxevVqrrjiiqNe2+FwMHv2bDZs2MCgQYO48sormTp1Ku3b/6dnV0R46623GD9+PNdffz19+/bl8ssvZ8+ePQBkZmby4YcfcsIJJ3DppZfSv39/brzxRiorg19GXC4Xr7zyCoWFhQwZMoQbb7yR+++/v+5259Fcd911/OQnP+Gqq65iyJAhfPbZZ9x7770H7XPOOecwd+7culGqI0aM4IUXXsDtPvjpzmuvvZaampqY3dIEkOb8DMPw4cN1yZKmrUNoTKKrensdxeNnkvfIWCpeWI5vfQltF/0U94D8eIcWFyKyVFUjPsuuiFwEjFXVn4Y+/wQYGX4LU0RaA2WqWi0i1wGXqOqZIvJLIF1Vfxva7y6gUlX/t77rNab9qlqyiZqvd5BzxWmIw3pKI2X16tX0798/3mGYBHfrrbcyf/58li078vO/hzraz1VD2y975syYBJc2thfu73Rg/9R3wCG0fnNiyiZmUbYD6Bz2uVOorI6qFod9fAZ4KOzYMYcc+0GkAguUVuHITrPEzJgY2r9/P2vXruXpp5+OyfQZ4ey2pjEJTkTIve9MEMj9w9mkj+8T75CS1WKgt4h0FxEPcCkwJ3wHEQl/Gn8CUDvb6DzgHBFpKSItgXNCZRHh7pmPZ3CXSJ3OGNMA559/PmeccQYXXHABl19+eUyvbT1nxjQD6ef2ol3RrThbZ8Y7lKSlqj4RuYlgUuUEZqjqShH5DbBEVecAN4vIBMAHlABXho4tEZH7CCZ4AL9R1cZPc14Pd5fWkTqVMaaBPvjgg7hd25IzY5oJS8yiT1XnAnMPKbs77P3twO31HDsDmBHVAI0xKcFuaxpjjElZzXlQnEk8kfp5suTMGGNMSnK73XXTNhgTCZWVlYdNxXE8LDkzxhiTkvLz89mxYwcVFRXWg2aaRFWpqKhgx44ddSsbNIU9c2aMMSYl5ebmAsHlgRqz5qIxR+J2uykoKKj7uWoKS86MMcakrNzc3Ij8MTUmkuy2pjHGGGNMArHkzBhjjDEmgVhyZowxxhiTQCw5M8YYY4xJIJacGWOMMcYkEGnOc7uISBGwpRGHtAH2RCmcRGN1TU6pUtej1bOrqraNZTDRcJT2K1X+H9dKtfpC6tU51eoL9de5Qe1Xs07OGktElqjq8HjHEQtW1+SUKnVNlXoeSarVPdXqC6lX51SrLzS9znZb0xhjjDEmgVhyZowxxhiTQFItOXs63gHEkNU1OaVKXVOlnkeSanVPtfpC6tU51eoLTaxzSj1zZowxxhiT6FKt58wYY4wxJqGlRHImImNFZI2IrBeR2+IdT1OJyAwRKRSRr8PKWonIfBFZF/q3ZahcROTPobqvEJGh8Yu88USks4i8LyKrRGSliPwiVJ509RWRdBH5XES+DNX116Hy7iLyWahOr4iIJ1SeFvq8PrS9WzzjPx4i4hSRZSLyz9DnpK3rsSRbO1UrldorSK02q1Yqtl0Q3fYr6ZMzEXEC04BxwABgoogMiG9UTfY8MPaQstuA91S1N/Be6DME69079JoC/CVGMUaKD/h/qjoAOBm4MfT/LxnrWw2cqaqDgZOAsSJyMvB74GFV7QXsBa4J7X8NsDdU/nBov+bmF8DqsM/JXNd6JWk7Vet5Uqe9gtRqs2qlYtsF0Wy/VDWpX8ApwLywz7cDt8c7rgjUqxvwddjnNUD70Pv2wJrQ+6eAiUfarzm+gDeAs5O9vkAm8AUwkuBEhq5Qed3PMzAPOCX03hXaT+IdeyPq2IngH6kzgX8Ckqx1bcB/i6Rsp8Lqk5LtVagOKdFmhcWf9G1XKO6otl9J33MGdAS2hX3eHipLNgWquiv0/lugIPQ+aeof6goeAnxGktY31E2+HCgE5gMbgH2q6gvtEl6furqGtu8HWsc24iZ5BLgVCIQ+tyZ563oszfrn9jgk5e/voVKhzaqVYm0XRLn9SoXkLOVoMD1PqmG4IpINvAZMVdUD4duSqb6q6lfVkwh+KxsB9ItzSFEhIj8AClV1abxjMfGVTL+/4VKlzaqVKm0XxKb9SoXkbAfQOexzp1BZstktIu0BQv8Whsqbff1FxE2wkZupqq+HipO2vgCqug94n2DXeAsRcYU2hdenrq6h7XlAcYxDPV6nARNEZDMwi+CtgUdJzro2RFL83DZCUv/+pmKbVSsF2i6IQfuVCsnZYqB3aBSFB7gUmBPnmKJhDjA59H4yweccasuvCI0IOhnYH9a1nvBERIBngdWq+qewTUlXXxFpKyItQu8zCD6nsppgQ3dRaLdD61r73+Ai4N+hb+QJT1VvV9VOqtqN4O/kv1V1EklY1wZKlXaqVtL9/tZKpTarViq1XRCj9iveD9XF6MG98cBagvfA74x3PBGoz8vALsBL8L72NQTvX78HrAPeBVqF9hWCo8A2AF8Bw+MdfyPrOopg9/8KYHnoNT4Z6wsMApaF6vo1cHeovAfwObAemA2khcrTQ5/Xh7b3iHcdjrPeY4B/pkJdj/HfIanaqbB6pUx7FapDyrRZYXVOybYrVJeotF+2QoAxxhhjTAJJhduaxhhjjDHNhiVnxhhjjDEJxJIzY4wxxpgEYsmZMcYYY0wCseTMGGOMMSaBWHJmjDHGGJNALDkzCU1E7hWRv8U7DmOMaSxrv8zxsuTMGGOMMSaBWHJmEoaI/EpEdohIqYisEZHvA3cAl4hImYh8GdovT0SeFZFdof1/KyLO0LYrReQTEXlcRPaLyDciclY862WMSX7WfplIch17F2OiT0T6AjcB31HVnSLSDXACDwC9VPXysN2fJ7hocC8gC/gnsA14KrR9JPAq0Aa4EHhdRLqrakn0a2KMSTXWfplIs54zkyj8QBowQETcqrpZVTccupOIFBBcp26qqparaiHwMMHFZ2sVAo+oqldVXwHWAN+PfhWMMSnK2i8TUdZzZhKCqq4XkanAvcBAEZkH3HKEXbsCbmCXiNSWOQh+86y1Qw9eNHYL0CHiQRtjDNZ+mciznjOTMFT1JVUdRbABU+D3oX/DbQOqgTaq2iL0ylXVgWH7dJSwlg/oAuyMZuzGmNRm7ZeJJEvOTEIQkb4icqaIpAFVQCUQAHYD3UTEAaCqu4B/AX8UkVwRcYhITxEZHXa6fOBmEXGLyI+B/sDcmFbIGJMyrP0ykWbJmUkUacCDwB7gW4IN1O3A7ND2YhH5IvT+CsADrAL2Enx4tn3YuT4DeofOdT9wkaoWR7sCxpiUZe2XiSg5+Na2Mc2biFwJ/DR0e8EYY5oNa79MLes5M8YYY4xJIJacGWOMMcYkELutaYwxxhiTQKznzBhjjDEmgVhyZowxxhiTQCw5M8YYY4xJIJacGWOMMcYkEEvOjDHGGGMSiCVnxhhjjDEJ5P8Df4DXYU4YuL0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(runner, 'dotproduct-loss-acc.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.4.2 模型评价\n",
    "\n",
    "模型评价加载最好的模型，然后在测试集合上进行评价。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluate on test set, Accuracy: 0.86936\n"
     ]
    }
   ],
   "source": [
    "model_path = \"checkpoint/model_best.pdparams\"\n",
    "runner.load_model(model_path)\n",
    "accuracy, _ =  runner.evaluate(test_loader)\n",
    "print(f\"Evaluate on test set, Accuracy: {accuracy:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从上面的实验可以看出：\n",
    "\n",
    "（1）在不加注意力机制的情况下，测试集上的准确率为0.86064，加入了加性注意力后，测试集的准确率为0.86488；换成点积注意力后，测试集上的准确率为0.86936。\n",
    "相比于不加注意力机制的模型，加入注意力机制的模型效果会更好些。\n",
    "\n",
    "（2）另外，从加性注意力和点积注意力的结果可以看出，点积注意力的准确率更好些。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.1.4.3 注意力可视化\n",
    "\n",
    "为了验证注意力机制学到了什么，我们把加性注意力的权重提取出来，然后进行可视化分析。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_path = \"checkpoint/model_best.pdparams\"\n",
    "model_atten = Model_LSTMAttention(\n",
    "    hidden_size,\n",
    "    embedding_size,\n",
    "    vocab_size,\n",
    "    n_classes=n_classes,\n",
    "    n_layers=n_layers,\n",
    "    use_additive=False,\n",
    ")\n",
    "# runner.load_model(model_path)\n",
    "model_state_dict = paddle.load(model_path)\n",
    "model_atten.set_state_dict(model_state_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "输入的文本为：this great science fiction film is really awesome\n",
      "转换成id的形式为：[[  10   88 1196 1839   24    7   61 1635]]\n",
      "训练的注意力权重为：[[0.09350643 0.25606698 0.08582163 0.11344668 0.09067672 0.09059495\n",
      "  0.09276529 0.17712136]]\n"
     ]
    }
   ],
   "source": [
    "text = \"this great science fiction film is really awesome\"\n",
    "# text = \"This movie was craptacular\"\n",
    "# text = \"I got stuck in traffic on the way to the theater\"\n",
    "# 分词\n",
    "sentence = text.split(\" \")\n",
    "# 词映射成ID的形式\n",
    "tokens = [\n",
    "    word2id_dict[word] if word in word2id_dict else word2id_dict[\"[oov]\"]\n",
    "    for word in sentence\n",
    "]\n",
    "# 取前max_seq_len的单词\n",
    "tokens = tokens[:max_seq_len]\n",
    "# 序列长度\n",
    "seq_len = paddle.to_tensor([len(tokens)])\n",
    "# 转换成Paddle的Tensor\n",
    "input_ids = paddle.to_tensor(tokens, dtype=\"int64\").unsqueeze(0)\n",
    "inputs = [input_ids, seq_len]\n",
    "# 模型开启评估模式\n",
    "model_atten.eval()\n",
    "# 设置不求梯度\n",
    "with paddle.no_grad():\n",
    "    # 预测输出\n",
    "    pred_prob = model_atten(inputs)\n",
    "# 提取注意力权重\n",
    "atten_weights = model_atten.attention.attention_weights\n",
    "print(\"输入的文本为：{}\".format(text))\n",
    "print(\"转换成id的形式为：{}\".format(input_ids.numpy()))\n",
    "print(\"训练的注意力权重为：{}\".format(atten_weights.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:101: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead\n",
      "  ret = np.asscalar(ex)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABOYAAABkCAYAAAAiyz/qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu81FW9//HXWxQ0L4hXClCxMETzSnjpqHhMxEywzMIyQU3U9KedTjdPpmYns+xUv1NqUIrlqbCjXXZeUgMx1DAwr2AoIgUomaBQoCD6OX+sNfBlmL1nZrM3w977/Xw85jEz67u+l5lZM9/1/cy6KCIwMzMzMzMzMzOzjWuzRh+AmZmZmZmZmZlZV+TAnJmZmZmZmZmZWQM4MGdmZmZmZmZmZtYADsyZmZmZmZmZmZk1gANzZmZmZmZmZmZmDeDAnJmZmZmZmZmZWQM4MGdmZmZmZmZmZgZIGi5ptqQ5kr7QQr6TJYWkwYW0i/N6syUdV8v+Nm+LgzYzMzMzMzMzM+vIJHUDrgGOBRYA0yU1RcSssnzbAhcBDxXSBgGjgH2AtwG/k7RXRLzR0j7dYs7MzMzMzMzMzAyGAHMiYm5ErAImAiMr5PsK8HXgtULaSGBiRKyMiOeAOXl7LWrzFnPLJkyNtt6mdV6bf+TgRh+CdTCrb3640YdgHcyrv3+y0YdgHchWR+7b6EMwMzNbY9WcRY0+BOtgdvrqKWr0MWyKXnti/ppY1Vb77XYOMLaweHxEjM+P+wDzC8sWAIcUtyXpIKBfRNwu6bOFRX2AaWXr9ql2bO7KamZmnZaDcmZmZmZmVpSDcOOrZqxA0mbAt4AxbXU8DsyZmZmZmZmZmZnBQqBf4XnfnFayLbAvMEUSQG+gSdKIGtatyGPMmZmZmZmZmZmZwXRggKT+krqTJnNoKi2MiKURsVNE7BERe5C6ro6IiBk53yhJPST1BwYAf6y2QwfmzMzMzMzMzMysy4uI1cAEYDawHFgUETMlXZFbxSHpXElPSHoUOBDon1dfDgwClgFPAU9Vm5EVHJgzMzMzMzMzMzNDUjfS+HEDga2B3pIGRcSlEVFqOffTiHhXRBwAfBw4q7CJ2RHRIyK2jIgTatmnA3NmZmZmZmZmZmYwBJgTEXMjYhUwERhZzBARywpPtwaCDeDAnJmZmZmZmZmZdQmSxkqaUbiNLSzuA8wvPF+Q08q3cb6kZ4FvABcWFvWX9Iik+yQdUcvxeFZWMzMzMzMzMzPrtLbab7c1jyNiPDB+Q7YXEdcA10j6KHAJMBp4AdgtIhZLOhj4laR9ylrYrcct5szMzMzMzMzMzGAh0K/wvG9Oa85E4CSAiFgZEYvz44eBZ4G9qu3QgTkzMzMzMzMzMzOYDgyQ1F9Sd2AU0FTMIGlA4ekJwDM5fec8eQSS9gQGAHOr7dCBOTMzMzMzMzMz6/IiYjUwAZgNLAcWRcRMSVdIGpGzjZP0mqRXgZuAL+f0I4HnJa0EZgHjImJJtX06MGdmZmZmZmZmZl1ebvE2BhhImnG1t6RBEXFpRJRazp0UEVtGxFbA6cD5Of0pYBGwHbA3MLrUgq4lDsyZmZmZmZmZmZnBEGBORMyNiFWkMeRGFjOUTeawNRD58UhgYh5r7jlgTt5eizwrq5mZmZmZmZmZGfQB5heeLwAOKc8k6Xzg00B34F8L604rW7dPtR26xZyZmZmZmZmZmXVay5cvX3OTNFbSjMJtbL3bi4hrIuLtwOeBSzbk2NxizszMzMzMzMzMuoSIGA+Mb2bxQqBf4XnfnNacicB1rVwXcIs5MzMzMzMzMzMzgOnAAEn9JXUHRgFNxQySBhSengA8kx83AaMk9ZDUHxgA/LHaDt1izszMzMzMzMzMuryIWC1pAjAbEDA5ImZKugKYkWdmvV7SENKkD68CH87rzpT0LmBZXvZoRLxRbZ8OzJmZmZmZmZmZWZcnqRswBhhImrxhuqRBEXFpIdtlwEMRsULSecDZwO/yshURsU09+3RXVjMzMzMzMzMzMxgCzImIuRGxijSG3Mhihoi4NyJW5KfTSGPJtZoDc2ZmZmZmZmZmZtAHmF94viCnNecs4M7C8y3zTK/TJJ1Uyw7dldXMzMzMzMzMzDotPfHC2seHvmMsMLaweHyeqbW+bUqnAYOBowrJu0fEQkl7ApMlPRERz7a0HQfmzMzMzMzMzMysS8hBuOYCcQuBfoXnfXPaOiS9F/gicFRErCxse2G+nytpCnAg0GJgzl1ZzczMzMzMzMzMYDowQFJ/Sd2BUUBTMYOkA4FxwIiIeLGQ3ktSj/x4J+A9wKxqO6wpMCdpuKTZkuZI+kLNL8fMzMzMzMzMzKwDiIjVwARgNrAcWBQRMyVdIWlEznYzsDvwuKR/SLonp+8NzJG0ktTK7vcRseGBuTxV7DXA8cAg4FRJg+p8bWZmZmZmZmZmZpusHAMbAwwEtgZ6SxoUEZdGRKnl3DnAdhGxFfA5YElO/zPwOvBWoDdwtKRe1fZZS4u5qlPFmpmZmZmZmZmZdXBVY2ARcW9ErMhPp5HGoQM4DrgnIpZExMvAPcDwajusJTBX71SxZmZmZmZmZmZmHU29MbCzgDtbuS7QRpM/SBoraYakGROmNFVfwczMzMzMzMzMbCNY3vSnNbdiDCvfxrZmm5JOAwYDV2/IsW1eQ56qU8UWp5pdNmFqbMgBmZmZmZmZmZmZtYdiDKuCqjEwAEnvBb4IHBURKwvrDi1bd0q146mlxVzVqWLNzMzMzMzMzMw6uKoxMEkHAuOAERHxYmHRXcAwSb3ypA/DclqLqgbm8lSxF+SNPQX8PCJm1viCzMzMzMzMzMzMNnk5BjYBmA0sBxZFxExJV0gakbP9ANgNeFrSc5Ka8rpLgN2BRfn2Sk5rUS1dWYmIO4A76n1BZmZmZmZmZmZmHYGkbsAYYCBp8obpkgZFxKWFbB8CtgM+AzRFxC2FZSsiYpt69llTYM7MzMzMzMzMzKyTGwLMiYi5AJImAiOBWaUMETEvL3uzLXbYJrOympmZmZmZmZmZdXB9gPmF5wtyWq22zDO9TpN0Ui0ruMWcmZmZmZmZmZl1Wm88/9Kax5LGAmMLi8fnmVrbwu4RsVDSnsBkSU9ExLMtreDAnJmZmZmZmZmZdQk5CNdcIG4h0K/wvG9Oq3XbC/P9XElTgAOBFgNz7spqZmZmZmZmZmYG04EBkvpL6g6MAppqWVFSL0k98uOdgPdQGJuuOQ7MmZmZmZmZmZlZlxcRq4ELgLuAp4CfR8RMSVdIGgEg6d2SFgCnAOMkzcyr7w3MkPQYcC9wVUQ4MGdmZmZmZmZmZlajN4HItzcAIuLSiCi1nNsKeBHoAZwTEfvkPA8C3wTeAmwJrK5lZw7MmZmZmZmZmZlZlyepG3ANcDwwCDhV0qCybH8FxgA/LVt3B+Ay4BBgCHCZpF7V9unAnJmZmZmZmZmZWQqozYmIuRGxCpgIjCxmiIh5EfE4qWVd0XHAPRGxJCJeBu4BhlfboWdlNTMzMzMzMzOzTmurI/dd81jSWGBsYfH4PFMrQB9gfmHZAlILuFpUWrdPtZUcmDMzMzMzMzMzsy4hB+HGV824kbgrq5mZmZmZmZmZGSwE+hWe981p7bauA3NmZmZmZmZmZmYwHRggqb+k7sAooKnKOiV3AcMk9cqTPgzLaS1yYM7MzMzMzMzMzLq8iFgNTABmA8uBRRExU9IVkkYASDpc0grgdOBnkp7Oq29HGlNuUb7Ni4gl1fbpwJyZmZmZmZmZmXV5kroBY4CBwNZAb0mDIuLSiCi1nDsA+HFEbAZ8HHiksImnI6JHvh1dyz4dmDMzMzMzMzMzM4MhwJyImBsRq4CJwMiyPCOBH+XHtwDHSFJrd+hZWc3MzMzMzMzMrNP6x5mT1jzueeaRY4GxhcXj80ytkLqizi8sWwAcUra5NXkiYrWkpcCOeVl/SY8Ay4BLImJqtWNzYM7MzMzMzMzMzLqEHIQbXzVj/V4AdouIxZIOBn4laZ+IWNbSSu7KamZmZmZmZmZmBguBfoXnfXNaxTySNgd6AosjYmVELAaIiIeBZ4G9qu3QgTkzMzMzMzMzMzOYDgyQ1F9Sd2AU0FSWpwkYnR9/CJgcESFp5zx5BJL2BAYAc6vt0F1ZzczMzMzMzMysy8tjxk0AZgMiBd1mSroCmJFnZr0JmClpFbAKOD6vfiRwraTtgQCujIgl1fbpFnNmZmZmZmZmZtbl5RZvY4CBwNZAb0mDIuLSHJQDOA24PSK6A58ALsjpTwGLgO2AvYHRpRZ0LXFgzszMOq2tjty30YdgZmZmZmYdxxBgTkTMjYhVwERgZFmekcCP8uNbgGMkKadPzGPNPQfMydtrUZt3Zd3ujCPU1tvsLCSNLUzBa9Yil5dmnHFEo49gk+UyU9l2LjMVubxYvVxmrB4uL1Yvlxmrh8uL1atPXL4mViVpLDC2sHh8oTz1AeYXli0ADinfXClP7vq6FNgxp08rW7dPtWNzi7mNa2z1LGZruLxYvVxmrB4uL1Yvlxmrh8uL1ctlxurh8mKtFhHjI2Jw4dbQIK8Dc2ZmZmZmZmZmZrAQ6Fd43jenVcwjaXOgJ7C4xnXX48CcmZmZmZmZmZkZTAcGSOovqTswCmgqy9MEjM6PP0SauTVy+ihJPST1BwYAf6y2wzYfY85a5D7wVg+XF6uXy4zVw+XF6uUyY/VwebF6ucxYPVxerF3kMeMuAO4CugE3RMRMSVcAM/LMrNcDN0maAywhBe/I+X4OzAJWA+dHxBvV9qkU1DMzMzMzMzMzM7ONyV1ZzczMzMzMzMzMGsCBOTMzMzMzMzMzswbo0oE5SUMlhaTL61jn8rzO0PY7MjNrT5KmSHI/fqtI0oWSZkl6Nf/efyrfT2nHfd6Y97FHe+3DOq5ay6TrKNYeJO2Ry9WNjT4W23T4N8jMrO10+sCcKxO2KWtNcNjM2o+kUcD/B14DvgN8GZjWBtv1xYq1SnuVSTMzMzPbNHhW1vp9D5gI/LXRB2JmrXY68JZGH4Rtkt5fuo+I50uJkvYGVrTjfi8GrgIWtuM+rGNqVJk0K1kI7A0sbfSBmJmZdUYOzNUpIl4CXmr0cZhZ60WEA+vWnLcBFAMg+fmf23OnEfEC8EJ77sM6rIaUSbOSiHgdcHkzMzNrJ526K2vuHvhcfjo6dyMq3caU5T1A0u2SXpG0QtJ9kg6vtM1K3ZEkHSHpN5IWSFopaZGkaZIua6eXZxUouSiPxfOapIWSviepp6R5kuYV8o4plQVJw/O4Y0vLxx6TNDCP/zRf0ipJf5P0U0nvrLD/vSRdJWmGpL/nsvAXSeMl9S3LeyNwb356WVn5HNrmb04HJmmEpEmSXsjv6fP5O/rJsnw7SPqqpCfz93ippMfyZ7J1IV+zY8xJOk7SHZJeyvt6VtLVkravkHdevm2d8/w1rzNH0uclqZl9DJF0cy6fK/PrulvShyvkPUTSLfk3ZVUuh+Mkva3+d9KaU/ptB47Oz9d8HwvPp1RYr5ukcyU9kMvbq/nz/6GkATnPPKB0Lri3fNs5T7NjzEn6sKTfF7b/hKSLJfWokLdVZdI2Pa0tk81sK/Lv3q6SbsjnseWSHpR0RM5TKjN/yWVmpqRT2uv1WcehCsPC5LL0TUmzc1l6JT++UdKeDTzcLqH4mSjVPW+W9KKkN5XrkLlO9DVJT+Vzx1KlutSwCtvrKemzkiYrXcusUqrHNkk6bAOOs5dSfezZFupEv8mvZXBr97MpULqeuFXS3Px+L8t1g9PK8v0sv94BZek/yumTytK3lfS6pN9X2Oepku7N37/X8md9STP1g5qvVSW9VdI1uT5RKgu/kHRwM6+7dD11rKSpkv6Z15mgXH+WdKCk2yS9nJc3qZlxdespu2adRWdvMTcF2B64CHgM+FVh2aN5GcBg4HPAH4AfArsBJwOTJB0QEbNb2omk4cDtwDKgidTkfwdSs/9PksaDsY3jGuA84HlgPLAKGAEMAbYAXq+wzoeA4cCdwPeB3UsL8mf7i7zub4A5QF/gg8AJko6OiD8VtvVB4FxSwO3BvP99gE8AJ0oaHBGlrmql8jgauI9UXkvm1f3KOylJY4FxwCLSZ/ASsAuwH3AGcG3O15/0vu8OPAxcR/rzYS/g30if7fIq+7oMuBxYAtwGvJj38xngfZIOi4hlZattAdxFatVyJ7AaOInULXFLyr7/ks7Ox/YG6ffimfx6BpN+L35eyHsmqRyvzHnnAwNYW54Odeu/NjMl348hlaGqv9uSupPKybGkz+anpPPAHsAHgPtJn+93SGXiKOBH1PH9lnQlqZvrS3n7/wSOB64EjpM0LCJWla1WV5m0TdaUfD+GGstkFdsDDwD/AH5GqqeMAu7KF97jctptpDJ0KnCzpPkR4THtbA1JbyGVpbcD95DOzSKV05HALcDchh1g1/J24CHgaeAnwFbAMkm7k35D9gCmAr8FtiZ1jf+tpHMi4geF7ewNfBX4Pema5mXS9dAI4HhJJ0bEb+s9uIh4WdJEUn3tvaTysoakfqRz2sMRMaPe7W9irgNmkt7DF4AdgfcBN0l6Z0R8KeebRPrtPYZURyg5Jt8fLmnLiHgtPz+KdM1eHrC7gfS+LgBuBV4BDgW+Ahwj6diIWJ3z1nytmuvT95PqEJNJ54t+wCmka5+TI+K2Cq9/BKl83Uaqcx9OOn/tIenifPxTgeuBdwEnAntK2i8i3izsv96ya9Y5RESnvpG+1AHcWGHZ0LwsgDFly87J6deWpV+e04cW0m7NaftX2MdOjX4PusoNOCJ/DrOB7Qvp3UknyQDmFdLH5LQ3geEVtteLVDF5CRhUtmxf0gXyn8rS+wA9KmxrGCkQc10zZfDyRr9/m+qNFGRbCexSYdlOhccP5vfy4kr5gC0Lz6ekn7918hyd13+wWH7Kysq3y9Ln5fQ7gK0K6buQKkivAFsU0geRgsNLgH0qHGffwuO9SIHdOUCfsnzH5PL0y0Z/Pp3tVqls5PQAppSlXZnTm8q/90APYOfC8/XOHWX5b8zL9yikHZbT/gr0LqRvTroQDuA/NqRM+rbp3+oskxXLGWvrOt8HNiukfzynL8llqvg7WTqn+nemi98oq0uTLqjXOyfmZd2BbRt9zJ39VvhMAriywvIppPrtqLL07UmNE14Fdi2k96TCNQvpz+jngacqLKvpN4j0x2MAt1TYRin/2Y1+T9vgM3l7hbTupIDU6+S6HLBnfs3/W8j3zpx2d74/prDs2zntiELamJz2i+K5vuw9vaiQVvO1KumPvQC+WJZ+OOmPvsXANhWOZTVwVCF9M1IgtnSO+VjZ9q7Py0ZuSNn1zbfOcuvUXVnr8EBE3FiWdgPpB2ZIHdt5tTwh0ph0tnGMzvdfjYhXSomRWpNc3MJ6v47K/wKeTjoJXBYRs4oLIuJJ4AfAgZIGFdIXRsTK8g1FxN2kf9GOq/XF2DpWU6G1Y+n7lZvWH0Y6YX+9Ur5Y+89jcy7M92cXy09e/8a87Y81t25EvFrI/yLwa1JFt9jl+TxSUOUrETGzwnEuKMu7BalitbAs3yRSMOhESdtWeV3WDiR1I/3L/Cpwbvn3PiJWRsTfN3A3Z+b7/4yIRYVtrwb+nVRx/UQz69ZaJq3rWAF8NgotE0itMFeT/oi6qPg7GRFTSYHeAzbmQVqHUqneuyoi/tGIg+mi/sb6LfP3J7WyujUiJhaX5frNZaTW0ycX0pdWumbJ9ZJbgIGSdmvNAUZqCTcDGCmpd+E4uwFnsbYVb4cWEc9WSFtF6s2zOblFXETMJf22Hl3o3ltqLXcp6Y/XYwqbOYbU46PYcvki0m/3mcVzffYVUvCsUp21xWtVpWF3hpH+EPxGWb4HWdva+oMVtv2ziLivkP9N4Kb89MmI+ElZ/h/n+zXnmNaUXbPOorN3Za3Vek2nI+J1SX8jVVar+QnpB+ohSTeTutM9UHaRbe3vwHx/f4Vl00gnsEr+2Ex6aUyN/ZXGKyy3V77fG5gFaYw70olwDLA/qfx0K6xT3uXMqvsJ8F/ArNwd4j7S96sY9Dg0399VdtFZj8NIwb9TVHlcpe7AzpJ2jIjFhfSlETGnQv75+b74G1I6zjtrPB6AoyS9u8LyXUhlay9Sq0LbuAaSglwPRdmg/G3ooHw/uXxBRDwtaQHQX1LPiCjOllhPmbSu4+nygElEvJHrOlvni8VyC4FDNsrRWUdyH6lsfEHSQaQWug8Aj0bEGw09sq7nsQp/CJfqDz2bqb/unO/3LiZKeg8p4HMYqY7RvWy9PqSATWtcS2r0cCaptTmkbp59Sb1J/tnK7W4ycuDy86RA2m6kbsVFfQqPJ5PeiwOAR4B/BV6IiGmSHs7bQNLOpF46d0eahKXUlXx/Uo+eT6ny0H0rWffzrfVatXQtNbW0vzKTgdNyvh+XLavUFblUP6pUTy396Vwcg7tVZdesM3BgLnmlmfTVrBtUqSgifiHp/aQWDGeSusGSf1gvjoh7Wlrf2kzPfP+38gX54mNxeXq2qJn0HfP92VX2u03h8beAT5HGlriLdNIp/Ts1hsL4dVabiPiWpJdIrZMuJL2/Iek+UuuPGawdL3JhM5upxY6k38TLquTbhvRPZElLvx+w7m9IPcdZKn+freF4bONrizJXTek3rbnZWl8gVf63B4qBuXrKpHUdS5tJX11lmeuKto6IWCbpUFJLrRGs7Q3wkqRrSa18K13UW9urVIct1R+OzbfmrKk/SPoAqWXca6Tuh8+SWmm9SRp25SjSEA2tNZH0J+vZkq7Kf6KOzcvGbcB2NwlKE578kfTH11RSl9SlpNZve5B69RTfv0mka8ZjJD1GGk7ljsKyz0nqSQrYiXXHl+uV03amep0VqOtatZZ6B6ytAxVVOo+srmHZFoW0usuuWWfhylYbiYjbgduVZn48hDRA5XnAbZIOLO8Kae2iNCj/rpQNOpyby+9I5YvoaGZ7pZPI/hHxeLWdS9qFFDh6Eji8vGWCpFOrbcMqi4gfAz/OMzsdThpY/0zSoOUDWRuI6NPMJmqxlDT20g4bdLAtKx7nn2s4HoCesf6EE9Z4bVHmqimVgd6ki6Ryby3LZ2a2UeSWNmflngKDSAGE80ld8TYDvtTC6tZ2KtVhS+eEiyLiv2vczldIvToGR8RTxQWSxpECc60WEa8qzer7b8AwSTNJkz48FBGPbci2NxGfJl1nnFE+PFKu/48uy19qCf/e/HgH1gbfJpOG4DmatV1aiy3nS5/vIxFxEDWq8Vq1WO+opL3rHa0pu2adQlcYY67UpH6jtBCIiOURMTkiPk1qqt2ddOKx9vdIvv+XCssOpf5AdGkshyNqzL8n6Tt1d4WgXN+8vNxGLZ8dXUS8EhF3RMTZpMHydwCOZO1ndZyk1v6uTQN6Sdpnw4+0xX1Abb8J9ZY/27j+TArO7SfpbTXkb813vfSbNrR8gaR3kLp/PFc+JqKZ2cYSycyI+C5rW7ic1MhjslbVH94BzKoQlNuMyvXq1riOFEg8hzS2XDc6QWu57B35/tYKy9YLauZxY2eRPqPhObkUmHuA1BX1GFLA+2XW1gfI3X5nAvtIqvvP5CrXqmuupSRVum46Ot//qd791sh1X+uyukJg7mXSSaBVA5bWQtKRzfx47ZrvV7TXvm0dpbEOvpibfwMgqTtrx7OoxwTShfdlktabBETSZpKGFpLm5ft/yS30Svm2IU0UUamMlLpEtlv57OgkFQfHLdol36+IiIdJs6keQBrfo3wbO0rassquvp3vf1Ap0CJp69x1Z0NcR2q6/6XipCGFfRTH2fgeacy7b0vaq0Le7pJccWmQPI7StaQxZL4vaZ0uPvnz2bmQ1Jrv+g35/pLitvLvyzdJ5/Dr6z12M7MNIWkfSbtWWOR67yYgD/ExFfigpDMr5ZH0rtzTo2QeMKBY/8l1r8tJLSLb4rieIQWf3g+cS6pjT2xxpY5jXr4fWkyUdBzNT9I0GXgLaVy/ZyJiPqTWhcAfgA8DbyfNfFs+fvK3SAG1G3JvknVI6pXHfyw9r+laNbeEvYfU/fZTZds8BPgo6dr6l828pg3SyrJr1il0+q6sEfFPSQ8BR0j6CfA0qeVCUxvu5r+BPpIeIP0wrwIOJv3L8Rc6z0lnkxYR90kaTxqzYqakW0mBjRNJTaOfJ42VUev2Fkv6EOnkM03SJNI/VAH0Iw1QuiNpdiAiYlGenGAU8Kiku0ljNRxLGrPjUdaf3W42qXvtKEmvk8pLADdFxF/qfxc6pV8C/5Q0jfT9EumftHeTBpP9Xc53GmmK9SslnZwfCxhAmmFqIGsrTuuJiEmSvgB8DXhG0h3Ac6RxLHYn/eN5P2v/2axbRMyS9Eng+8Ajkn4NPEMqR+8mdcc+Ouf9c66U3EAqz78l/X5tQQruHAH8Pb8ua4wvk7qDnAg8Lek20uxy/Uhl7rOklp2QBlp+E/iapH1JFVsi4j+b23hEPCjpG8DngCcl3UIa8+d40mDQ9wNXt/3LMjNr0bHA1ZL+QDovvUhqwTuS9Dvn36XG+ygp8HO9pAuBh0iBsL7AfqRzyGGkzw7Sn5Olukmp/vweUlDuN6TzXFu4ltR9c1fguxVmFO2orgXOAP43n6ufJ73Hw4GfAx+psM4k4ALSH82/qLBsaOHxOiLiBkkHk8ZfflbSXaSJOXYA+pN6k0wgBUChvmvVc0mt9q6WNIw0qUM/4BTS9/uMdp55ud6ya9YpdPrAXPZx0glnOHAq6WJ9AS1cpNfpStKYV4NJJ5s3ST+OVwLfiYiX22g/Vt15pC5m55BOLItJgZ3/IH3mlcZpalYO1uwHfIY0uPERpJPZ86STRnmT9bNI49t9hDTWyt9JQeBLK+QtTUrxAeAq0glvW1L5vJ90ojT4Aum9P4g0g9drpPfm86SZvF4HiIjn8r+DnyN1o7kg551HGnC46gk8Ir6eKy0XkrpujCQFdRcC44GfbuiLiYgfSHqSVKaG5mN9CXgc+GFZ3v/JgwL/OylgN4wUmHmeNEjzzRt6PNZ6EbFK0nDSb83ppDFkRPp8fklhhuiIeErSaNLn/klyQB9oNjCX1/u8pEdI5fl0UmD2WeAS4L8iwjM9m9nGdhfpD6IjSefJ7UiDwt/iBheCAAABnklEQVQDfCsiHmzgsRmp5VMO3Pw/4GTgY6Suo6UulN8FnijkHydpJamV1GjSxGVTScGmk2m7wFwTqc6zE52nGysR8biko0nn9BNI19iPkWZCfYXKgbkppGvGzVh/9vVJpHH/qLCstM/zJd1JqoO8lzQhwxLSNejVwP8Ustd8rRoRcyUNJtUz3keqqy4Dfgt8NSKmt/hmbKB6y65ZZ6GI5sa9N+s8JA0g/as7MSI8CYOZmZmZ2UaUZy+dAzwQER6Ow8ws6wpjzFkXIql3+eD/kt4CfCc/bZcxEczMzMzMrEWfIbUs/16jD8TMbFPiFnPWqUi6itRdeQqpW0Vv0qxGfYE7gRPChd7MzMzMrN1J2o00btgAUtfYx4GDKkxoYGbWZXWVMeas67gH2J80FtcOpBkwnyYNevodB+XMzMzMzDaaPUkTa60g1dPPc1DOzGxdbjFnZmZmZmZmZmbWAB5jzszMzMzMzMzMrAEcmDMzMzMzMzMzM2sAB+bMzMzMzMzMzMwawIE5MzMzMzMzMzOzBnBgzszMzMzMzMzMrAH+D/xSRIzWRM2tAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x108 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "from matplotlib import rcParams\n",
    "rcParams.update({'figure.autolayout': True})\n",
    "# 对文本进行分词，得到过滤后的词\n",
    "list_words = text.split(\" \")\n",
    "# 提取注意力权重，转换成list\n",
    "data_attention = atten_weights.numpy().tolist()\n",
    "# 取出前max_seq_len变换进行特征融合，得到最后个词\n",
    "list_words = list_words[:max_seq_len]\n",
    "# 把权重转换为DataFrame，列名为单词\n",
    "d = pd.DataFrame(data=data_attention, columns=list_words)\n",
    "f, ax = plt.subplots(figsize=(20, 1.5))\n",
    "# 用heatmap可视化\n",
    "# sns.heatmap(d, vmin=0, vmax=0.4, ax=ax)\n",
    "# sns.heatmap(d, vmin=0, vmax=0.4, ax=ax, cmap=\"OrRd\")\n",
    "\n",
    "my_colors=['#e4007f', '#f19ec2', '#e86096', '#eb7aaa', '#f6c8dc', '#f5f5f5', '#000000', '#f7d2e2']\n",
    "sns.heatmap(d, vmin=0, vmax=0.4, ax=ax, cmap=my_colors)\n",
    "# 纵轴旋转360度\n",
    "label_y = ax.get_yticklabels()\n",
    "plt.setp(label_y, rotation=360, horizontalalignment=\"right\")\n",
    "# 横轴旋转0度\n",
    "label_x = ax.get_xticklabels()\n",
    "plt.setp(label_x, rotation=0, horizontalalignment=\"right\", fontsize=20)\n",
    "plt.savefig('att-vis.pdf')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "输出结果如图所示，颜色越深代表权重越高，从图可以看出，注意力权重比较高的单词是\"great\"，\"awesome\"。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "********\n",
    "实现《神经网络与深度学习》第8.2节中定义的其它注意力打分函数，并重复上面的实验。\n",
    "********"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 8.2 . 基于双向LSTM和多头自注意力的文本分类实验\n",
    "\n",
    "在上一节介绍的注意力机制中需要一个外部的查询向量$\\mathbf q$，用来选择和任务相关的信息，并对输入的序列表示进行聚合。\n",
    "在本节中，我们进一步实现更强大的自注意力模型，同样和双向LSTM网络一起来实现上一节中的文本分类任务。\n",
    "\n",
    "## 8.2.1 自注意力模型\n",
    "\n",
    "当使用神经网络来处理一个变长的向量序列时，我们通常可以使用卷积网络或循环网络进行编码来得到一个相同长度的输出向量序列。基于卷积或循环网络的序列编码都是一种局部的编码方式，只建模了输入信息的局部依赖关系。虽然循环网络理论上可以建立长距离依赖关系，但是由于信息传递的容量以及梯度消失问题，实际上也只能建立短距离依赖关系。\n",
    "\n",
    "\n",
    "**自注意力** {Self-Attention}是可以有效的直接解决长程依赖问题的方法，相当于构建一个以输入序列中的每个元素为单元的全连接网络（即每个全连接网络的每个节点为一个向量），利用注意力机制来“动态”地生成全连接网络的权重。\n",
    "\n",
    "下面，我们按照从简单到复杂的步骤分别介绍简单自注意力、QKV自注意力、多头自注意力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.2.1.1 最简单的自注意力\n",
    "\n",
    "我们先来看最简单的自注意力模型。\n",
    "\n",
    "假设一个输入序列$\\mathbf X\\in \\mathbb{R}^{L\\times D}$，为了建模序列中所有元素的交互关系，我们将可以将输入序列中每个元素$\\mathbf x_m$作为查询向量，利用注意力机制从整个序列中选取和自己相关的信息，就得到了该元素的上下文表示$\\mathbf h_m\\in \\mathbb{R}^{1\\times D}$。\n",
    "$$\n",
    "\\mathbf h_{m} = \\sum_{n=1}^{L} \\alpha_{mn} \\mathbf x_n =softmax(\\mathbf x_m \\mathbf X^T) \\mathbf X,\n",
    "$$\n",
    "其中$\\alpha_{mn}$表示第$m$个元素对第$n$个元素的注意力权重，注意力打分函数使用点积函数。\n",
    "\n",
    "输入一个文本序列The movie is nice进行，如果计算单词movie的上下文表示，可以将movie作为查询向量，计算和文本序列中所有词的注意力分布，并根据注意力分布对所有词进行加权平均，得到movie的上下文表示。和卷积网络或循环网络相比，这种基于注意力方式会融合更远的上下文信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/bd809b713ff14dae8c30f08279865004a28045a292b845fd8178cca6ae2922e2\" width=\"800px\"></center>\n",
    "<br><center>图8.7 自注意力示例</center></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "对于输入序列$\\mathbf X\\in \\mathbb{R}^{L\\times D}$，自注意力可以表示为\n",
    "$$\n",
    "\\mathbf Z = softmax(\\mathbf X \\mathbf X^T) \\mathbf X,\n",
    "$$\n",
    "其中$softmax()$是按行进行归一化，Z表示的是注意力分布的输出。计算方式如下图所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/5add2e1686d0436a8be1e79270442daa27c807661b0a4695a4bd6d81023111e9\" width=\"400px\"></center>\n",
    "<br><center>图8.8 整个序列的自注意力示例</center></br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "********\n",
    "动手实现上面的简单注意力模型，并加入到第8.1节中构建模型的LSTM层和注意力层之间，观察是否可以改进实验效果。\n",
    "********"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.2.1.2 QKV自注意力\n",
    "\n",
    "上面介绍的简单自注意力模型只是应用了注意力机制来使用序列中的元素可以长距离交互，模型本身不带参数，因此能力有限。\n",
    "\n",
    "为了提高模型能力，自注意力模型经常采用**查询-键-值**{Query-Key-Value，QKV}模式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "将输入序列$\\mathbf X\\in \\mathbb{R}^{B\\times L\\times D}$经过线性变换分别映射到查询张量、键张量、值张量。\n",
    "\n",
    "$$\n",
    "\\qquad \\mathbf Q=\\mathbf X\\mathbf W^{Q} \\in \\mathbb{R}^{B\\times L \\times D},\\\\\n",
    "\\qquad \\mathbf K=\\mathbf X\\mathbf W^{K} \\in \\mathbb{R}^{B\\times L \\times D},\\\\\n",
    "\\qquad  \\mathbf V=\\mathbf X\\mathbf W^{V} \\in \\mathbb{R}^{B\\times L \\times D},\n",
    "$$\n",
    "其中$\\mathbf W^{Q} \\in \\mathbb{R}^{D \\times D}$,$\\mathbf W^{K} \\in \\mathbb{R}^{D \\times D}$,$\\mathbf W^{V} \\in \\mathbb{R}^{D \\times D}$是可学习的映射矩阵。为简单起见，这里令映射后$\\mathbf Q,\\mathbf K,\\mathbf V$的特征维度相同，都为$D$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "QKV自注意力的公式表示如下：\n",
    "$$\n",
    "\\mathrm{attention}(\\mathbf Q,\\mathbf K,\\mathbf V) =  softmax(\\frac{\\mathbf Q\\mathbf K^T}{\\sqrt{D}}) \\mathbf V\n",
    "$$\n",
    "其中$softmax(\\cdot)$是按行进行归一化。计算方式如下图所示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/7008c957a13a406a9673397afb10b23cc3000efbe49e498f844b0bead85483c9\" width=\"400px\"></center>\n",
    "<br><center>图8.9 整个序列的自注意力示例</center></br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "QKV注意力中，假设$\\mathbf Q$和$\\mathbf K$都是独立的随机向量，都满足0均值和为1的单位方差，那么$\\mathbf Q$和$\\mathbf K$点积以后的均值为0，方差变成了$D$，当输入向量的维度$D$比较高的时候，QKV注意力往往有较大的方差，从而导致Softmax的梯度比较小，不利于模型的收敛，因此QKV注意力除以了一个$\\sqrt{D}$有效降低方差，加速模型收敛。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**屏蔽序列中的[PAD]**\n",
    "\n",
    "在QKV注意力的实现中，需要注意的如何屏蔽[PAD]元素不参与注意力的计算。\n",
    "\n",
    "带掩码的QKV注意力实现原理示意如下图所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/9f96ef66626749a1b8afb9f44d209effa10daabfa3c04ac9affe75cf7be53d5d\" width=\"800px\"></center>\n",
    "<br><center>图8.10 QKV注意力中屏蔽[PAD]位置的实现过程示意</center></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "具体实现步骤如下：\n",
    "\n",
    "（1）根据序列的长度创建一个掩码张量$\\mathbf M\\in \\{0,1\\}^{B\\times L \\times L}$，每个序列都有一个真实的长度，对于每个序列中小于真实长度的位置设置为True，对于大于等于真实长度的位置，说明是填充[PAD]，则设置为False。\n",
    "```\n",
    "# arrange: [1,seq_len],比如seq_len=4, arrange变为 [0,1,2,3]\n",
    "arrange = paddle.arange((seq_len), dtype=paddle.float32).unsqueeze(0)\n",
    "# valid_lens : [batch_size*seq_len,1]\n",
    "valid_lens = valid_lens.unsqueeze(1)\n",
    "# mask [batch_size*seq_len, seq_len]\n",
    "mask = arrange < valid_lens\n",
    "```\n",
    "\n",
    "\n",
    "（2）根据布尔矩阵mask中False的位置，将注意力打分序列中对应的位置填充为-inf(实际实现过程中可设置一个非常小的数，例如-1e9)。\n",
    "\n",
    "```\n",
    "# 给mask为False的区域填充-1e9\n",
    "# y: [batch_size, seq_len, seq_len]\n",
    "y = paddle.full(score.shape, -1e9, score.dtype)\n",
    "# score: [batch_size, seq_len,seq_len]\n",
    "score = paddle.where(mask, score, y)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "QKV自注意力的代码实现如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F\n",
    "import paddle.nn as nn\n",
    "import paddle\n",
    "\n",
    "class QKVAttention(nn.Layer):\n",
    "    def __init__(self, size):\n",
    "        super(QKVAttention, self).__init__()\n",
    "        size = paddle.to_tensor([size], dtype=\"float32\")\n",
    "        self.sqrt_size = paddle.sqrt(size)\n",
    "\n",
    "    def forward(self, Q, K, V, valid_lens) :\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - Q：查询向量，shape = [batch_size, seq_len, hidden_size]\n",
    "            - K：键向量，shape = [batch_size, seq_len, hidden_size]\n",
    "            - V：值向量，shape = [batch_size, seq_len, hidden_size]\n",
    "            - valid_lens：序列长度，shape =[batch_size]\n",
    "        输出：\n",
    "            - context ：输出矩阵，表示的是注意力的加权平均的结果\n",
    "        \"\"\"\n",
    "        batch_size, seq_len, hidden_size = Q.shape\n",
    "        # score: [batch_size, seq_len, seq_len]\n",
    "        score = paddle.matmul(Q, K.transpose((0, 2, 1))) / self.sqrt_size\n",
    "        # arrange: [1,seq_len],比如seq_len=2, arrange变为 [0, 1]\n",
    "        arrange = paddle.arange((seq_len), dtype=paddle.float32).unsqueeze(0)\n",
    "        # valid_lens : [batch_size*seq_len, 1]\n",
    "        valid_lens = valid_lens.unsqueeze(1)\n",
    "        # mask [batch_size*seq_len, seq_len]\n",
    "        mask = arrange < valid_lens\n",
    "        # mask : [batch_size, seq_len, seq_len]\n",
    "        mask = paddle.reshape(mask, [batch_size, seq_len, seq_len])\n",
    "        # 给mask为False的区域填充-1e9\n",
    "        # y: [batch_size, seq_len, seq_len]\n",
    "        y = paddle.full(score.shape, -1e9, score.dtype)\n",
    "        # score: [batch_size, seq_len,seq_len]\n",
    "        score = paddle.where(mask, score, y)\n",
    "        # attention_weights: [batch_size, seq_len, seq_len]\n",
    "        attention_weights = F.softmax(score, -1)\n",
    "        self._attention_weights = attention_weights\n",
    "        # 加权平均\n",
    "        # context: [batch_size, seq_len, hidden_size]\n",
    "        context = paddle.matmul(attention_weights, V)\n",
    "        return context\n",
    "\n",
    "    @property\n",
    "    def attention_weights(self):\n",
    "        return self._attention_weights\n",
    "\n",
    "paddle.seed(2022)\n",
    "Q = paddle.rand(shape=[2, 2, 3])\n",
    "K = paddle.rand(shape=[2, 2, 3])\n",
    "V = paddle.rand(shape=[2, 2, 3])\n",
    "valid_lens = paddle.to_tensor([1, 1, 2, 2])\n",
    "print(\"查询向量为 {}\".format(Q.numpy()))\n",
    "print(\"键向量为 {}\".format(K.numpy()))\n",
    "print(\"值向量为 {}\".format(V.numpy()))\n",
    "qkv_atten = QKVAttention(3)\n",
    "context = qkv_atten(Q, K, V, valid_lens)\n",
    "print(\"注意力的输出为 : {}\".format(context.numpy()))\n",
    "print(\"注意力权重为 : {}\".format(qkv_atten.attention_weights.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出可以看出，QKV注意力的输入是$\\mathbf Q$，$\\mathbf K$，$\\mathbf V$，然后计算缩放点积注意力得到最终的输出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.2.1.3 多头自注意力\n",
    "\n",
    "为了进一步提升自注意力的能力，我们将QKV注意力扩展为**多头**{Multi-Head}模式，其思想和多通道卷积非常类似，利用多组QKV自注意力来提升模型能力。\n",
    "\n",
    "**多头自注意力**{Multi-Head Self-Attention，MHSA}首先会分别进行多组的QKV注意力的计算，其中每组称为一个**头**{head}。每单个头可以看作是序列中所有元素的一次特征融合。之后，把得到的多个头拼接到一起，通过线性变换进行特征融合，得到最后的输出表示。多头自注意力的结构如图8.11所示，分为三个部分：线性变换、单头QKV注意力和多头融合。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/96d9403101e94dd9b2691fc9f91c5c331364b9a31ffc42cf8cb7aa81154dbc8e\" width=\"800px\"></center>\n",
    "<br><center>图8.11 多头注意力结构图</center></br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "假设输入序列为$\\mathbf X\\in \\mathbb{R}^{B\\times L\\times D}$，其中$B$为批量大小，$L$为序列长度，$D$为特征维度，通过以下三个步骤来实现多头自注意力。\n",
    "\n",
    "（1）**线性变换**：分别计算每个单头的QKV张量。在计算第$m$个头时，将$\\mathbf X$做三个线性变换分别映射到分别查询张量、键张量和值张量：\n",
    "$$\n",
    "\\qquad \\mathbf Q_m=\\mathbf X\\mathbf W_{m}^{Q}\\\\\n",
    "\\qquad \\mathbf K_m=\\mathbf X\\mathbf W_{m}^{K}\\\\\n",
    "\\qquad  \\mathbf V_m=\\mathbf X\\mathbf W_{m}^{V}\n",
    "$$\n",
    "其中$\\mathbf W_{m}^{\\mathbf Q} \\in \\mathbb{R}^{D \\times D_m}$,$\\mathbf W_{m}^{K} \\in \\mathbb{R}^{D \\times D_m}$,$\\mathbf W_{m}^{\\mathbf V} \\in \\mathbb{R}^{D \\times D_m}$，$D_m$是映射后QKV的特征维度。\n",
    "\n",
    "（2）**单头QKV自注意力**：分别计算每个单头的QKV自注意力。通过QKV自注意力计算第$m$个头$head_{m}$：\n",
    "$$\n",
    "head_{m}= \\mathrm{attention}(\\mathbf Q_m,\\mathbf K_m,\\mathbf V_m) \\in \\mathbb{R}^{B\\times L\\times D_m},\n",
    "$$\n",
    "其中$\\mathrm{attention}(\\cdot)$为公式QKV自注意力中的函数。\n",
    "\n",
    "（3）**多头融合**：将多个头进行特征融合：\n",
    "$$\n",
    "\\mathbf Z=\\mathrm{MultiHeadSelfAttention}(\\mathbf X)\n",
    "\\triangleq\n",
    "\\bigoplus(head_{1},head_{2},...,head_{M})\\mathbf W'\n",
    "$$\n",
    "其中$\\oplus$表示对张量的最后一维进行向量拼接，$\\mathbf W' \\in \\mathbb{R}^{(MD_m) \\times D'}$是可学习的参数矩阵，$D'$表示的输出特征的维度。\n",
    "\n",
    "这里为了简单起见，令每个头的特征维度$D_m=\\frac{D}{M}$，输出特征维度$D'=D$和输入维度相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**动手实现**\n",
    "\n",
    "下面动手实现多头自注意力。为了能够使多个头的QKV自注意力可以并行计算，需要对QKV的张量进行重组，其实现原理如图8.12所示。\n",
    "\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/76e448f3ce6f4fe88380b285436c8c34bca9dc36131842d582fa09f6caba1ea0\" width=\"800px\"></center>\n",
    "<br><center>图8.12 多头自注意力实现原理图</center></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "具体实现步骤如下：\n",
    "\n",
    "（1）**线性变换**：将输入序列$\\mathbf X\\in \\mathbb{R}^{B\\times L\\times D}$做线性变换，得到$\\mathbf Q,\\mathbf K,\\mathbf V \\in \\mathbb{R}^{B\\times L\\times D}$三个张量。这里使用`nn.Linear`算子来实现线性变换，并且得到的$\\mathbf Q,\\mathbf K,\\mathbf V$张量是一次性计算多个头的。\n",
    "\n",
    "```\n",
    "# 查询\n",
    "Q_proj = nn.Linear(qsize, inputs_size, bias_attr=False)\n",
    "# 键\n",
    "K_proj = nn.Linear(ksize, inputs_size, bias_attr=False)\n",
    "# 值\n",
    "V_proj = nn.Linear(vsize, inputs_size, bias_attr=False)\n",
    "batch_size, seq_len, hidden_size = X.shape\n",
    "# Q,K,V: [batch_size,seq_len,hidden_size]\n",
    "Q, K, V = Q_proj(X), K_proj(X), V_proj(X)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "（2）**多头分组**：上一步中得到的$\\mathbf Q,\\mathbf K,\\mathbf V \\in \\mathbb{R}^{B\\times L\\times D}$张量是一次性计算多个头的，需要对$\\mathbf Q,\\mathbf K,\\mathbf V$张量的特征维度进行分组，分为$M$组，每组为一个头。\n",
    "$$\n",
    "(B\\times L\\times D) \\xRightarrow {\\text{reshape}} (B\\times L\\times M \\times D_m),\n",
    "$$\n",
    "其中$M$是头数量，$D_m$是每个头的特征维度，并有$D=M\\times D_m$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "（3）**形状重组**：在上一步分组后，得到$\\mathbf Q,\\mathbf K,\\mathbf V \\in \\mathbb{R}^{B\\times L\\times M\\times D_m}$。由于不同注意力头在计算QKV自注意力是独立的，因此把它们看做是不同的样本，并且把多头的维度$M$合并到样本数量维度$B$，便于计算QKV自注意力。\n",
    "$$\n",
    "(B \\times L \\times M \\times D_m) \\xRightarrow{\\text{transpose}} (B \\times M \\times L \\times D_m) \\xRightarrow{\\text{reshape}} (BM \\times L \\times D_m)\n",
    "\n",
    "$$\n",
    "对每个$\\mathbf Q,\\mathbf K,\\mathbf V$都执行上面的操作，得到$\\mathbf Q,\\mathbf K,\\mathbf V \\in \\mathbb{R}^{((B M) \\times L \\times D_m)}$。\n",
    "\n",
    "经过形状重组后，$B$个样本的多头自注意力转换为$B*M$个样本的单头QKV自注意力。\n",
    "这里实现了`split_head_reshape`函数来执行上面第（2）、（3）步的操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def split_head_reshape(X, heads_num, head_size):\n",
    "    \"\"\"\n",
    "    输入：\n",
    "        - X：输入矩阵，shape=[batch_size, seq_len, hidden_size]\n",
    "    输出：\n",
    "        - output：输出多头的矩阵，shape= [batch_size * heads_num, seq_len, head_size]\n",
    "    \"\"\"\n",
    "    batch_size, seq_len, hidden_size = X.shape\n",
    "    # X: [batch_size, seq_len, heads_num, head_size]\n",
    "    # 多头分组\n",
    "    X = paddle.reshape(x=X, shape=[batch_size, seq_len, heads_num, head_size])\n",
    "    # X: [batch_size, heads_num, seq_len, head_size]\n",
    "    # 形状重组\n",
    "    X = paddle.transpose(x=X, perm=[0, 2, 1, 3])\n",
    "    # X: [batch_size*heads_num, seq_len, head_size]\n",
    "    X = paddle.reshape(X, [batch_size * heads_num, seq_len, head_size])\n",
    "    return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "（4）**QKV注意力**：对最新的$\\mathbf  Q,\\mathbf  K,\\mathbf  V \\in \\mathbb{R}^{(BM)\\times L \\times D_m)}$，计算QKV注意力：\n",
    "$$\n",
    "\\mathbf  H= \\mathrm{attention}(\\mathbf  Q,\\mathbf  K,\\mathbf  V) \\in \\mathbb{R}^{(BM)\\times L \\times D_m},\n",
    "$$\n",
    "其中$\\mathrm{attention}()$为公式QKV注意力中的函数。\n",
    "\n",
    "```\n",
    "attention = QKVAttention(head_size)\n",
    "# out: [batch_size*heads_num, seq_len, head_size]\n",
    "out = attention(Q, K, V, valid_lens)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "（5）**重组恢复**：将形状重组恢复到原来的形状\n",
    "\n",
    "$$\n",
    "(B M \\times L \\times D_m) \\xRightarrow{\\text{reshape}} (B \\times M \\times L \\times D_m) \\xRightarrow{\\text{transpose}}(B \\times L \\times M \\times D_m)\n",
    "$$\n",
    "\n",
    "```\n",
    "# out: [batch_size, heads_num, seq_len, head_size]\n",
    "out = paddle.reshape(out, [batch_size, heads_num, seq_len, head_size])\n",
    "# out: [batch_size, seq_len, heads_num, head_size]\n",
    "out = paddle.transpose(out, perm=[0, 2, 1, 3])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "（6）**多头融合**：根据公式\\eqref{eq:att-mhsa-output-linear}将多个头进行特征融合。首先将多头的特征合并，还原为原来的维度。\n",
    "\n",
    "$$\n",
    "(B\\times L\\times M \\times D_m) \\xRightarrow{\\text{reshape}}(B\\times L\\times D),\n",
    "$$\n",
    "\n",
    "然后再进行特征的线性变换得到最后的输出。这里线性变换也使用\\code{nn.Linear}实现。\n",
    "\n",
    "```\n",
    "# 输出映射\n",
    "out_proj = nn.Linear(inputs_size, inputs_size, bias_attr=False)\n",
    "\n",
    "# 多头注意力输出拼接\n",
    "# out: [batch_size, seq_len, heads_num * head_size]\n",
    "out = paddle.reshape(x=out, shape=[0, 0, out.shape[2] * out.shape[3]])\n",
    "# 输出映射\n",
    "out = self.out_proj(out)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**多头自注意力算子**\n",
    "\n",
    "多头自注意力算子`MultiHeadSelfAttention`的详细代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn as nn\n",
    "import paddle\n",
    "\n",
    "class MultiHeadSelfAttention(nn.Layer):\n",
    "    def __init__(self, inputs_size, heads_num, dropout=0.0):\n",
    "        super(MultiHeadSelfAttention, self).__init__()\n",
    "        # 输入的embedding维度\n",
    "        self.inputs_size = inputs_size\n",
    "        self.qsize, self.ksize, self.vsize = inputs_size, inputs_size, inputs_size\n",
    "        # head的数目\n",
    "        self.heads_num = heads_num\n",
    "        # 每个head的输入向量的维度\n",
    "        self.head_size = inputs_size // heads_num\n",
    "        # 输入的维度inputs_size需要整除head数目heads_num\n",
    "        assert (\n",
    "            self.head_size * heads_num == self.inputs_size\n",
    "        ), \"embed_size must be divisible by heads_num\"\n",
    "        # 查询\n",
    "        self.Q_proj = nn.Linear(self.qsize, inputs_size, bias_attr=False)\n",
    "        # 键\n",
    "        self.K_proj = nn.Linear(self.ksize, inputs_size, bias_attr=False)\n",
    "        # 值\n",
    "        self.V_proj = nn.Linear(self.vsize, inputs_size, bias_attr=False)\n",
    "        # 输出映射\n",
    "        self.out_proj = nn.Linear(inputs_size, inputs_size, bias_attr=False)\n",
    "        # QKV注意力\n",
    "        self.attention = QKVAttention(self.head_size)\n",
    "\n",
    "    def forward(self, X, valid_lens):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "            - X：输入矩阵，shape=[batch_size,seq_len,hidden_size]\n",
    "            - valid_lens： 长度矩阵，shape=[batch_size]\n",
    "        输出：\n",
    "            - output：输出矩阵，表示的是多头注意力的结果\n",
    "        \"\"\"\n",
    "        self.batch_size, self.seq_len, self.hidden_size = X.shape\n",
    "        # Q,K,V: [batch_size, seq_len, hidden_size]\n",
    "        Q, K, V = self.Q_proj(X), self.K_proj(X), self.V_proj(X)\n",
    "        # Q,K,V: [batch_size*heads_num, seq_len, head_size]\n",
    "        Q, K, V = [\n",
    "            split_head_reshape(item, self.heads_num, self.head_size)\n",
    "            for item in [Q, K, V]\n",
    "        ]\n",
    "        # 把valid_lens复制 heads_num * seq_len次\n",
    "        # 比如valid_lens_np=[1,2],num_head*seq_len=2 则变为 [1,1,2,2]\n",
    "        valid_lens = paddle.repeat_interleave(\n",
    "                valid_lens, repeats=self.heads_num * self.seq_len, axis=0)\n",
    "        # out: [batch_size*heads_num, seq_len, head_size]\n",
    "        out = self.attention(Q, K, V, valid_lens)\n",
    "        # out: [batch_size, heads_num, seq_len, head_size]\n",
    "        out = paddle.reshape(\n",
    "            out, [self.batch_size, self.heads_num, self.seq_len, self.head_size]\n",
    "        )\n",
    "        # out: [batch_size, seq_len, heads_num, head_size]\n",
    "        out = paddle.transpose(out, perm=[0, 2, 1, 3])\n",
    "        # 多头注意力输出拼接\n",
    "        # out: [batch_size, seq_len, heads_num * head_size]\n",
    "        out = paddle.reshape(x=out, shape=[0, 0, out.shape[2] * out.shape[3]])\n",
    "        # 输出映射\n",
    "        out = self.out_proj(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "对上面的实现进行验证，输入以形状为$2\\times 2 \\times 4$的张量，表示两个2序列，每个序列有2个单词，每个单词的长度是4维。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "paddle.seed(2021)\n",
    "X = paddle.rand((2, 2, 4))\n",
    "valid_lens= paddle.to_tensor([1,2])\n",
    "print('输入向量 {}'.format(X.numpy()))\n",
    "multi_head_attn = MultiHeadSelfAttention(heads_num=2,inputs_size=4)\n",
    "context = multi_head_attn(X,valid_lens)  \n",
    "print('注意力的输出为 : {}'.format(context.numpy()))\n",
    "print('注意力权重为 : {}'.format(multi_head_attn.attention.attention_weights.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "多头自注意力把输入矩阵分成了2个头，分别计算QKV 注意力，计算结束后把2个头的结果拼接在一起输出。权重的输出为0的地方表示被掩码去除，不参与注意力的计算，所以输出为0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.2.2 基于LSTM和多头自注意力的文本分类的模型构建\n",
    "\n",
    "基于LSTM和多头自注意力的文本分类模型的模型结构如下图所示，整个模型由以下几个部分组成：\n",
    "\n",
    "1） 嵌入层：将输入句子中的词语转换为向量表示；\n",
    "\n",
    "2） LSTM层：基于双向LSTM网络来建模句子中词语的上下文表示；\n",
    "\n",
    "3） 自注意力层：使用多头自注意力机制来计算LSTM的自注意力特征表示；\n",
    "\n",
    "4） 汇聚层：对多头自注意力的输出进行平均汇聚得到整个句子的表示；\n",
    "\n",
    "5） 线性层：输出层，预测对应的类别得分.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/bb328336d6cf43d898eb09ee9f38315180908bffad0a4ba28f6dd7cdb5ada099\" width=\"800px\"></center>\n",
    "<br><center>基于双向LSTM和多头自注意力的文本分类模型</center></br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "本节中，我们直接复用第6.4节中实现的嵌入层和双向LSTM层，使用上一小节中定义的多头自注意力算子`MultiHeadSelfAttention`，以及第8.1.2.5节模型的线性层。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 8.2.2.1 模型汇总\n",
    "\n",
    "基于双向LSTM和多头注意力机制的网络就是在原有的双向LSTM的基础上实现了多头自注意力，求平均后接入分类层输出，代码实现如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class AveragePooling(nn.Layer):\n",
    "    def __init__(self):\n",
    "        super(AveragePooling, self).__init__()\n",
    "    \n",
    "    def forward(self, sequence_output, sequence_length):\n",
    "        sequence_length = paddle.cast(sequence_length.unsqueeze(-1), dtype=\"float32\")\n",
    "        # 根据sequence_length生成mask矩阵，用于对Padding位置的信息进行mask\n",
    "        max_len = sequence_output.shape[1]\n",
    "        mask = paddle.arange(max_len) < sequence_length\n",
    "        mask = paddle.cast(mask, dtype=\"float32\").unsqueeze(-1)\n",
    "        # 对序列中paddling部分进行mask\n",
    "        sequence_output = paddle.multiply(sequence_output, mask)\n",
    "        # 对序列中的向量取均值\n",
    "        batch_mean_hidden = paddle.divide(paddle.sum(sequence_output, axis=1), sequence_length)\n",
    "        return batch_mean_hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Model_LSTMSelftAttention(nn.Layer):\n",
    "    def __init__(\n",
    "        self,\n",
    "        hidden_size,\n",
    "        embedding_size,\n",
    "        vocab_size,\n",
    "        n_classes=10,\n",
    "        n_layers=1,\n",
    "        attention=None,\n",
    "    ):\n",
    "        super(Model_LSTMSelftAttention, self).__init__()\n",
    "        # 表示LSTM单元的隐藏神经元数量，它也将用来表示hidden和cell向量状态的维度\n",
    "        self.hidden_size = hidden_size\n",
    "        # 表示词向量的维度\n",
    "        self.embedding_size = embedding_size\n",
    "        # 表示词典的的单词数量\n",
    "        self.vocab_size = vocab_size\n",
    "        # 表示文本分类的类别数量\n",
    "        self.n_classes = n_classes\n",
    "        # 表示LSTM的层数\n",
    "        self.n_layers = n_layers\n",
    "        # 定义embedding层\n",
    "        self.embedding = nn.Embedding(\n",
    "            num_embeddings=self.vocab_size, embedding_dim=self.embedding_size\n",
    "        )\n",
    "        # 定义LSTM，它将用来编码网络\n",
    "        self.lstm = nn.LSTM(\n",
    "            input_size=self.embedding_size,\n",
    "            hidden_size=self.hidden_size,\n",
    "            num_layers=self.n_layers,\n",
    "            direction=\"bidirectional\",\n",
    "        )\n",
    "        self.attention = attention\n",
    "        # 实例化聚合层，聚合层与循环网络章节使用一致\n",
    "        self.average_layer = AveragePooling()\n",
    "        # 定义分类层，用于将语义向量映射到相应的类别\n",
    "        self.cls_fc = nn.Linear(\n",
    "            in_features=self.hidden_size * 2, out_features=self.n_classes\n",
    "        )\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        input_ids, valid_lens = inputs\n",
    "        # 获取词向量\n",
    "        embedded_input = self.embedding(input_ids)\n",
    "        # 使用LSTM进行语义编码\n",
    "        last_layers_hiddens, (last_step_hiddens, last_step_cells) = self.lstm(\n",
    "            embedded_input, sequence_length=valid_lens\n",
    "        )\n",
    "        # 计算多头自注意力\n",
    "        last_layers_hiddens = self.attention(last_layers_hiddens, valid_lens)\n",
    "        # 使用聚合层聚合sequence_output\n",
    "        last_layers_hiddens = self.average_layer(last_layers_hiddens, valid_lens)\n",
    "        # 将其通过分类线性层，获得初步的类别数值\n",
    "        logits = self.cls_fc(last_layers_hiddens)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.2.3 模型训练\n",
    "\n",
    "实例化组装RunnerV3的重要组件：模型、优化器、损失函数和评价指标，其中模型部分传入的是多头自注意力，然后便可以开始进行模型训练。损失函数使用的是交叉熵损失，评价函数使用的是Accuracy，跟上一节的设置保持一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from paddle.optimizer import Adam\n",
    "from nndl import Accuracy, RunnerV3\n",
    "\n",
    "paddle.seed(2021)\n",
    "# 迭代的epoch数\n",
    "epochs = 2\n",
    "# 词汇表的大小\n",
    "vocab_size = len(word2id_dict)\n",
    "# lstm的输出单元的大小\n",
    "hidden_size = 128\n",
    "# embedding的维度\n",
    "embedding_size = 128\n",
    "# 类别数\n",
    "n_classes = 2\n",
    "# lstm的层数\n",
    "n_layers = 1\n",
    "# 学习率\n",
    "learning_rate = 0.001\n",
    "# 指定评价指标\n",
    "metric = Accuracy()\n",
    "# 交叉熵损失\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "# 指定评价指标\n",
    "metric = Accuracy()\n",
    "multi_head_attn = MultiHeadSelfAttention(inputs_size=256, heads_num=8)\n",
    "# 框架API的复现\n",
    "# multi_head_attn = nn.MultiHeadAttention(embed_dim=256, num_heads=8, bias_attr=False)\n",
    "# 实例化基于LSTM的注意力模型\n",
    "model_atten = Model_LSTMSelftAttention(\n",
    "    hidden_size,\n",
    "    embedding_size,\n",
    "    vocab_size,\n",
    "    n_classes=n_classes,\n",
    "    n_layers=n_layers,\n",
    "    attention=multi_head_attn,\n",
    ")\n",
    "# 定义优化器\n",
    "optimizer = Adam(parameters=model_atten.parameters(), learning_rate=learning_rate)\n",
    "# 实例化RunnerV3\n",
    "runner = RunnerV3(model_atten, optimizer, criterion, metric)\n",
    "save_path = \"./checkpoint/model_best.pdparams\"\n",
    "# 训练\n",
    "runner.train(\n",
    "    train_loader,\n",
    "    dev_loader,\n",
    "    num_epochs=epochs,\n",
    "    log_steps=10,\n",
    "    eval_steps=10,\n",
    "    save_path=save_path,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化观察训练集与验证集的损失变化情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nndl import plot\n",
    "\n",
    "plot(runner, 'att-loss-acc2.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 8.2.4 模型评价"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_path = \"checkpoint/model_best.pdparams\"\n",
    "runner.load_model(model_path)\n",
    "accuracy, _ =  runner.evaluate(test_loader)\n",
    "print(f\"Evaluate on test set, Accuracy: {accuracy:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "1）在不加注意力机制的情况下，测试集上的准确率为0.86064。\n",
    "加入了多头注意力后，准确率变为了0.86552。说明多头注意力能够起到信息筛选和聚合的作用。\n",
    "\n",
    "2）和第8.1节的点积注意力和加性注意力相比，多头自注意力的准确率介于加性注意力和点积注意力之间，因此同等条件下，多头自注意力模型的效果要优于普通的加性注意力模型，最终点积注意力的准确率最高。\n",
    "\n",
    "********\n",
    "前面上一节的注意力机制相比，多头自注意力模型引入了更多的参数，因此模型会更复杂，通常需要更多的训练数据才能达到比较好的性能。请思考可以从哪些角度来提升自注意力模型的性能？\n",
    "********\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "任取一条英文文本数据，然后使用模型进行预测，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = \"this movie is so great. I watched it three times already\"\n",
    "# 句子按照空格分开\n",
    "sentence = text.split(\" \")\n",
    "# 单词转ID\n",
    "tokens = [\n",
    "    word2id_dict[word] if word in word2id_dict else word2id_dict[\"[oov]\"]\n",
    "    for word in sentence\n",
    "]\n",
    "# 取max_seq_len\n",
    "tokens = tokens[:max_seq_len]\n",
    "# 长度\n",
    "seq_len = paddle.to_tensor([len(tokens)])\n",
    "# 转换成Tensor\n",
    "input_ids = paddle.to_tensor([tokens], dtype=\"int64\")\n",
    "inputs = [input_ids, seq_len]\n",
    "# 预测\n",
    "logits = runner.predict(inputs)\n",
    "# label词表\n",
    "id2label = {0: \"消极情绪\", 1: \"积极情绪\"}\n",
    "# 取最大值的索引\n",
    "label_id = paddle.argmax(logits, axis=1).numpy()[0]\n",
    "# 根据索引取出输出\n",
    "pred_label = id2label[label_id]\n",
    "print(\"Label: \", pred_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果看，这句话的预测结果是积极情绪，这句话本身的情感是正向的，说明预测结果正确。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "********\n",
    "尝试将LSTM层去掉，只是用注意力层重复上面实验，观察结果并分析原因。\n",
    "********"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
